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Dear  IT  Executive: 

Today’s  CIO  is  being  challenged  on  many  fronts,  from  cost  containment,  business  alignment, 
compliance,  competitive  pressures  in  managing  outsourced  IT  services,  and  security.  Many 
experienced  IT  practitioners  point  out  that  the  solution  to  virtually  all  these  issues  is  more  repeatable 
IT  processes  and  effective  controls.  However,  merely  understanding  this  does  not  necessarily  equate 
to  an  effective  plan  to  solve  the  problems,  and  may  create  more  questions  than  answers. 

After  all,  best  practices  are  already  documented  in  process  frameworks  such  as  ITIL.  And,  analysts 
unanimously  point  to  change  management  processes  as  a  must-have:  Gartner  Group  says  that 
“operational  change  management  is  a  prerequisite  to  providing  high  IT  service  quality.  It  is  not 
optional.” 

The  Information  Technology  Process  Institute  (ITPI)  in  conjunction  with  Tripwire  has  written  a  book  to 
help  IT  to  address  this  need,  showing  how  to  get  started  on  implementing  effective  change 
management  processes.  The  book  is  called  ‘The  Visible  Ops  Handbook:  Starting  ITIL  in  4  Practical 
Steps."  ITPI  met  with  hundreds  of  IT  organizations  to  identify  eight  IT  groups  with  the  highest  service 
levels,  best  security  and  best  efficiencies.  Visible  Ops  reflects  the  lessons  learned  about  how  these 
organizations  work  and  describes  a  control-based  entry  point  into  the  world  of  ITIL  that  others  can 
use  to  springboard  their  own  process  improvement  efforts.  In  other  words,  not  only  do  you  start  your 
process  improvement  journey,  but  it  helps  you  build  effective  compliance  controls  as  well. 

The  ITPI  studied  hundreds  of  organizations,  and  identified  the  common  characteristics  of  seven  high- 
performing  IT  organizations.  They  found  that  what  they  had  in  common  were  three  key  elements:  1) 

A  culture  of  change  management,  2)  A  culture  of  causality,  and  3)  a  relentless  desire  to  find  variance 
early,  before  it  results  in  a  potentially  catastrophic  outage. 

By  summarizing  how  the  high-performing  IT  organizations  work,  Visible  Ops  presents  practices  that 
make  sense,  and  can  be  implemented  in  any  organization.  For  novice  organizations,  Visible  Ops 
provides  useful  guidance  on  where  to  start  their  improvement  efforts.  For  more  mature  organizations, 
Visible  Ops  provides  a  framework  for  continual  improvement. 

For  more  information  about  ITPI  visit  them  online  at  www.itpi.org  or  call  541 .485.4051 .  For  more 
information  about  Tripwire  visit  them  online  at  www.tripwire.com  or  call  1 .800.874.7947. 

I  invite  you  to  read  on  and  enjoy  this  complimentary  issue  of  CIO  magazine  and  the  Visible  Ops 
handbook,  courtesy  of  Tripwire  partnering  with  ITPI,  and  CXO  Media  Inc. 
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How  did  80%  of  information 
become  100%  useless? 

What  if  information  could  find  its  way  in  and  out  of 
databases,  all  on  its  very  own?  With  the  Adobe 
Intelligent  Document  Platform,  it's  possible.  When  you 
combine  the  logic  of  XML  and  Adobe  PDF,  suddenly 
documents  are  smarter.  Unstructured  content  unifies  with 
structured  data.  And  information  intuitively  travels  where 
it's  needed,  safely  and  securely.  It's  simplicity  at  work. 
The  Intelligent  Document  Platform.  Better  by  Adobe: 


See  how  smarter  documents  are  working  for  other  companies  at  adobe.com/idp. 
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EMERGENCY  PLAN? 
ABSOLUTELY.  DEFINITELY. 


IBM,  the  IBM  logo,  the  On  Demand  logo  and  Express  Portfolio  are  registered  trademarks  or  trademarks  of  International  Business  Machines  Corporation  in  the  United 
States  and/or  other  countries.  Other  company,  product  and  sen/ice  names  may  be  trademarks  or  sen/ice  marks  of  others.  ©2004  IBM  Corporation.  All  rights  reserved. 


ibm.com/ondemand 


You  don’t  have  the  time  or  resources  to  waste  on  hypothetical 
problems  that  may  arise  someday.  Right?  Wrong.  It’s  an  on  demand 
world.  Downtime  can  hurt  any  business.  But  it  can  really  hurt  a 
mid-sized  one.  The  unexpected  will  happen.  And  when  it  does,  your 

entire  company  needs  to  respond.  “METEORS?  I’M  NOT  WORRIED.” 

What  about  blackouts?  Hackers?  Sudden  spikes  in  demand?  The 

world  changes  daily.  And  the  longer  you  take  to  react  and  recover, 

the  more  it’ll  cost  you.  IBM  and  IBM  Business  Partners  are  ideally 

placed  to  help.  Our  experience  and  insight  mean  we  can  help  you 

pinpoint  vulnerabilities,  address  potentially  critical  weaknesses  and 

then  formulate  a  comprehensive,  end-to-end  resiliency  strategy. 

SOUNDS  MORE  EXPENSIVE  THAN  THE  EMERGENCY.” 

It’s  not.  The  IBM  Express  Portfolio™ of  offerings  -  hardware,  software, 
services  and  financing  -  are  designed  specifically  for  mid-sized 
companies  (and  their  mid-sized  budgets).  IBM  Protection  Express, 
for  example,  can  provide  planning,  rapid  shipment  of  recovery 
equipment,  mobile  facility  options,  telephone  support  and  easy- 

to-order  upgrades.  66  I  FEEL  SAFER  ALREADY.” 

You’re  not  alone.  Thousands  of  companies  already  count  on  IBM  Express 
Portfolio  offerings  to  help  run  their  businesses  smoothly  and  securely. 

You  can  never  be  sure  what’s  coming,  but  you  can  make  sure  you’re 
ready  for  it.  For  more,  visit  ibm.com/businesscenter/expressportfolio 
IBM  EXPRESS  PORTFOLIO  -  BUILT  FOR  MID-SIZED  BUSINESSES. 
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Lillian  Vernon  President 
JONATHAN  SHAPIRO  wanted 
to  change  the  company  fast  but 
admits  he  failed  to  help  his 
employees  change  along  with  it 


Having  lived  through  a 
troubled  ERP  rollout  that 
cost  Hewlett-Packard 
$160  million,  CIO  and 
Executive  VP  of  Global 
Operations  GILLES 
BOUCHARD  says  you 
can  never  do  too  much 
contingency  planning. 
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executive  authority.  Yet  with  his  skill  for  reading  situations  and 
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PIUS  2  tickets  to  SC 


Date:  February  14,  2005 
San  Francisco 

Visit  ca.com/etrust/workshop 

for  information  and  to  register  to  win. 


2004  Computer  Associates  International,  inc.  iCA).  All  rights  reserved. 
NO  PURCHASE  NECESSARY  Visit  ca.com 'etrust/workshop  for  Official  Rules 
and  prize  details.  Must  register  by  January  5,  2005.  Must  be  21  or  older  to 
enter.  Void  outside  of  the  United  States,  in  Florida  and  where  prohibited. 


It  takes  an  integrated  security  solution  to  make  sure  the 
right  people  have  the  right  access  at  the  right  time. 

eTrusf  Identity  and  Access  Management  Solutions 

These  days,  a  vital  aspect  of  security  management  is  providing  customized 
levels  of  access  for  countless  employees  and  partners  while  also  protecting 
your  customers  from  identity  theft.  That’s  one  complicated  job-and  one  that 
can  be  made  much  easier  with  CA’s  eTrust  Identity  and  Access  Management 
(1AM)  Solutions.  They  enhance  security  and  reduce  costs  by  automating 
processes  and  enabling  self-administration,  in  addition  to  providing  policy-based 
cross-platform  protection  for  web,  mainframe,  and  application  resources 
enterprise  wide.  To  find  out  how  CA’s  1AM  solutions  can  improve  your  business, 
attend  one  of  our  workshops,  ca.com/etrust/workshop 
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TO  HAVE  YOUR  BACKUPS  PROTECTED, 
CALL  FOR  ONE  OF  OUR  VEHICLES. 


With  Iron  Mountain,  your  backup  data  can  be  transported  by  road  or  by  wire.  Our  Electronic  Vaulting  Service 
is  another  dependable  way  to  have  your  data  protected  off-site.  And  without  the  need  for  an  IT  person  at  your  branch 
or  remote  locations. 

Electronic  Vaulting  also  means  your  files  are  backed  up  continuously,  further  reducing  the  risk  of  data  loss  if 
a  system  has  to  be  restored.  It  also  standardizes  your  backup  process  across  all  of  your  locations. 

For  a  copy  of  our  guide  "Calculating  the  Cost  of  Downtime,”  visit  www.ironmountain.com/downtime  or  to  speak 
with  a  sales  representative  call  1-800-899-IRON. 


IRON  MOUNTAIN 


©  2004  Iron  Mountain  Incorporated.  All  Rights  Reserved.  Iron  Mountain  and  the  design  of  the  mountain  are  registered  trademarks  of  Iron  Mountain  Incorporated. 
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TRY  IT  ONLINE 

Put  the  Correct  Face  on  It 

Think  you  know  what  your  colleagues  and 
staff  are  thinking?  Think  again.  Facial  expres¬ 
sions  can  be  as  fleeting  as  a  half-second,  even 
though  they  are  the  clearest  indicator  of  what 
someone  is  feeling.  In  How  to  Be  a  Mind 
Reader  (Page  72),  take  the  quiz  "How  Well 
Can  You  Read  a  Face?”  then  go  online  for  an 
expanded,  interactive  version  that  lets  you 
practice  the  fine  art  of  catching  the  fleeting 
expression.  You  can  find  the  Micro-Expression 
Training  Tool  with  the  online  version  of  this 
article  or  at  www.cio.com/communication. 


ONLINE  SPECIALS 

Get  the  Inside  Scoop 

Our  Special  Reports  cover  the  big 
topics  that  you  must  keep  up  with 
in  order  to  grow  your  business  and 
your  career.  Check  out  CIO. corn’s 
Special  Reports  on: 

►  Compliance— How  to  play  by  the 
new  rules  of  Sarbanes-Oxley, 

HIPAA  and  the  Patriot  Act.  Go  to 
www.cio.com/compliance. 

►  Running  IT  Like  a  Business- 
Internal  marketing  resources, 
interactive  tools,  articles  and  more, 
Go  to  www.cio.com/ritlab. 

►  State  of  the  CIO— Research  and 
resources  that  focus  on  your  role, 
your  career,  your  future.  Go  to 
www.cio.com/state. 


ADD A COMMENT 

Put  a  Face  on  It 

James  Hoopes, 
a  business  ethics 
professor,  worries 
that  the  informa¬ 
tion  age  has  made 
employees  invi¬ 
sible  to  managers  who  only 
know  of  them  through  a  spread¬ 
sheet  (The  Dehumanized 
Employee,  Page  40).  Managers 
often  make  decisions  affecting 
employees  who  are  far  away 
geographically,  which  allows 
those  managers  to  ignore  the 
human  beings  affected  by  their 
decisions.  What  can  IT  man¬ 
agers  do  to  help  alleviate  such 
impersonal  working  conditions? 
Go  to  the  Add  a  Comment  box 
at  the  end  of  this  column  onl  ine 
to  voice  your  opinion. 


BETTER  THAN  EVER 

A  New  You  for  the  New  Year? 

Looking  for  a  new  job?  Want  to  make  the  most  of  your  current  position? 
Interested  in  networking  within  your  field?  Our  Career  Resources  area 
provides  late-breaking  job  postings,  career-related  Research  Centers, 
lists  of  networking  events  and  a  wealth  of  other  services  that  will  help 
you  take  control  of  your  career.  Goto  www.cio.com/career. 


CIO.COM’S  BLOGS 

Just  What  You’ve  Been  Missing 

Wonder  why  the  CEO  suddenly  started  asking  why  you  need  a  data  center  if  all 
you  need  is  the  data?  Guess  you  missed  last  month’s  BusinessWeek  article. 
Keep  up  with  Web  Editor  Sandy  Kendall’s  blog  What’s  the  CEO  Reading? 

Wantto  nipthe  phishing  scam  in  the  bud?  Technology  Editor  Chris 
Lindquist's  Tech  LinkLetter  has  a  link  to  the  Internet  Fraud  Complaint  Center. 
Find  our  blogs  at  www.cio.com/blogs. 
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EVERY  BUSINESS  SHOULD  HAVE  A  PLAN. 
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JASON  Warren,  Chief  Information  Officer, 
Aeneas  Internet  and  Telephone,  Jackson,  I  N 


Whether  it's  a  tornado,  cyber  attack  or  other  disaster,  every  business  needs  an  emergency  plan.  A  plan 
can  save  lives,  your  company’s  network  infrastructure  and  your  entire  business  -  at  little  or  no  cost  to 
your  company.  You  can’t  control  what  happens.  But  you  can  be  prepared.  Visit  www.ready.gov  for 
practical  steps  you  can  take  now  to  give  your  company  a  better  chance  of  survival. 
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Knowing  is  more  than  possessing  knowledge.  It's  about  being  aware,  being  in  control 
and  getting  things  right.  And  in  today's  tightly  regulated,  highly  litigious  environment, 
you  have  to  know  you're  in  complete  compliance  24  hours  a  day,  every  day. 

NetlQ  Security  Management  is  the  only  way  to  assure  compliance,  manage  risk 
and  secure  assets.  Our  knowledge-based  software  solutions  are  intelligent  and 
simple  to  use.  Only  NetlQ,  a  leader  in  systems  and  security  management,  gives  you 
the  assurance  of  knowing  that  your  enterprise  is  secure,  available  and  performing. 
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Knowing 

you're  in  compliance. 


Knowing  is  everything. 


www,  netiq.com/solutions/security 
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BABY  STEPS  TO  KM 


The  Oct.  15  “Less  for  Success”  article  puts  many  things  in  perspective  and  follows 
the  thinking  behind  the  Chinese  proverb,  “The  journey  of  a  1,000  miles  begins 
with  a  single  step.”  The  article  also  highlights  a  classic  case  of  confusing  IT  for 
and  information  management  with  knowledge  management. 

A  failure  to  convince  executives  at  Children’s  Hospital  in  Boston 
to  invest  in  a  half-million-dollar  content  management  system  should 
not  be  viewed  as  a  failure  of  knowledge  management,  but  rather  the 
beginning  of  it.  Knowledge  management  is  a  reality  check  and 
should  involve  working  with  people  within  the  organization  and 
understanding  what  it  takes  to  do  the  right  things.  It  requires  effective 
communication  with  the  stakeholders  and  securing  buy-in  from  top  management.  Beside  IT 
and  content  management,  KM  professionals  are  expected  to  deal  with  organizational  learning 
and  cultural  issues  to  ensure  that  knowledge  utilization  and  knowledge  sharing  do  take  place. 


Suliman  Hawamdeh  •  Professor  and  Program  Coordinator 
University  of  Oklahoma  •  Suliman@hawamdeh.net 


INNOVATE  AND  EXECUTE 

“Decision  Evolution”  (Oct.  1)  claims 
that  automated  decision  systems  will 
remove  many  entry-level  jobs.  However, 
it  would  also  create  higher  volumes  of 
new  entry-level  jobs.  Just  as  the  paper¬ 
less  office  theory  resulted  in  more 
paper  being  used  than  ever  before,  I 
believe  more  people  will  be  needed  in 
the  workplace  as  change  accelerates.  It  is 
simply  impossible  for  automation  to 
keep  pace  with  the  rate  of  change. 

Many  business  executives  have  begun 
focusing  on  their  armies  of  subject  matter 
experts  (SMEs).  These  SMEs  are  respon¬ 
sible  for  the  instructional  rules  that  deter¬ 
mine  organizational  productivity  and 
performance  capabilities.  A  blend  of  auto¬ 
mated  and  human  decisioning  technolo¬ 
gies  is  needed.  Our  future  is  dependent 
on  our  ability  to  be  innovative  and  exe¬ 
cute  quickly,  thus  safeguarding  every¬ 
one’s  right  to  work. 

Freddie  McMahon,  CEO  and  Founder 

Decisionality 
freddie@decisionality.  com 


MOTIVATING  FACTORS 

I  wish  being  a  successful  CIO  were  as 
simple  as  “They  Don’t  Care  Too  Much 
for  Money”  (Sept.  15)  suggests. 

There  will  always  be  strife  in  the  tech¬ 
nology  operation  of  a  dynamic  organi¬ 
zation.  Neither  money  nor  inclusion 
nor  praise  will  solve  that.  As  the  CIO, 
the  only  way  I  keep  my  best  is  by  know¬ 
ing  what  motivates  them  to  stay.  For 
some,  it’s  money.  For  others,  it’s  praise 
and  recognition.  Others  require  inclu¬ 
sion  in  decision  making.  When  you  fig¬ 
ure  out  what’s  important  to  each  high- 
performer,  keeping  them  will  be  easy. 

Mark  Ives,  CIO 
Alion  Science  and  Technology 
mark@ivesnet.com 

Employee  motivation  depends  more 
on  personality  type  than  anything 
else.  Each  type  is  motivated  by  differ¬ 
ent  things.  Drivers  like  to  be  in  charge: 
A  reward  for  them  may  be  a  project 
lead  position.  Intuitors  prefer  time  to 
work  on  new  ideas.  Feelers  may  enjoy 


a  company  luncheon  where  everyone 
is  having  fun.  Thinkers  sometimes  like 
a  nice  pat  on  the  back — and  maybe  a 
new  toy,  like  a  computer  upgrade  or 
bigger  monitor. 

Managers  are  spending  too  little 
time  getting  to  know  their  employees. 
If  you’re  a  manager  and  don’t  know 
what  excites  your  employees,  you’re  not 
doing  your  job. 

Blair  Christensen,  Consultant 
MicroWorks 
blarman@gmail.  com 

CORRECTION 

A  box  accompanying  the  Nov.  1  story, 
“One  Outsources,  the  Other  Doesn’t” 
contained  an  error  in  the  first  sentence, 
j  The  sentence  should  read,  “While  Sears’ 

|  net  income  was  halved,  its  stock  price 
was  sinking  due  to  increased  competi¬ 
tion.”  We  apologize  for  the  error. 


What  Do  You  Think? 


Send  your  thoughts  and  feedback  to 
letters@cio.com.  Letters  may  be  edited  for 
length  or  clarity.  For  a  link  to  the  articles 
mentioned,  go  to  www.cio.com/printlinks. 
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82%  of  the  Fortune  500 

USE  OUR  BUSINESS  INTELLIGENCE  SOFTWARE. 

Perhaps  you  should  too. 


— . 


— 


Why  do  82%  of  Fortune  500w  companies  use  Business  Objects  business  intelligence  software? 
Simple.  They  get  better  information.  Accurate  information.  Consistent  information.  Up-to-date 
information.  Actionable  information.  All  in  dashboards,  reports,  and  scorecards  that  allow 
you  to  track,  understand,  and  manage  your  business. 

More  than  26,000  organizations  of  all  sizes  use  our  software  to  unlock  the  power  of  information 
to  improve  enterprise  performance.  If  you  don't,  perhaps  it's  time  to  see  what  you're  missing. 


Business  Objects 

The  business  intelligence  standard. 
To  see  how  business  leaders  around  the  world  use  our  software,  visit  ►  www.businessobjects.com/leaders 
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App  developers  live  in  fantasyland 
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Even  if  everyone  knew  about  the  problem,  would  anyone  know  the  solution? 


Web-based  applications  are  critical  to  your  success. 
But  the  reality  is,  most  aren't  doing  their  job  as  intended. 

They're  compromised  by  performance  issues,  security 
fears  and  mushrooming  costs  that  have  nothing  to  do 
with  their  real  role  in  life,  and  everything  to  do  with 
trying  to  coexist  with  a  network  too  focused  on  con¬ 
nectivity  and  not  nearly  adaptable  enough. 


Little  wonder  there's  friction  between  a  lot  of  application 
developers  and  network  managers.  And  just  adding 
another  point  solution,  code  fix  or  new  addition  to  the 
server  farm  isn't  going  to  improve  things. 

What  you  need  is  a  true  solution.  One  that  you  can 
easily  implement  that's  not  only  the  answer  now,  but 
for  whatever  the  future  brings.  A  solution  that  makes 
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NETWORK  GUYS  COULDN’T 
DELIVER  PIZZA 
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the  network  aware  of  the  application  and  gives  it  the 
intelligence  to  interact  with  the  application.  And  is  so 
comprehensive  it  gives  you  complete  control  over  who 
gets  access  from  where  and  when,  and  can  actually 
identify  and  filter  application-level  cyber  attacks. 

It's  application  traffic  management  taken  to  the  next  level. 
An  approach  that  unifies  all  the  application  optimization, 


availability  and  security  you  need  in  one  cohesive  archi¬ 
tecture  you  can  customize  to  specific  business  policies. 

It's  something  that  could  have  only  come  from 
a  deep  understanding  of  both  the  network  and 
the  application.  Which  is  why  it's  only  from  F5. 

For  more  information,  call  866-879-4132  or 
visit  www.f5.com/ciotaxi. 
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Connect.  Any  Way  You  Want. 


On-Demand  Collaboration.  Only  from  Polycom 


It'll  take  the  demands  off  you 


In  this  real-time  world,  instant  access  to  colleagues  around  the  globe  has  become  a  business  mandate.  Only  Polycom  can  bring 
people  together  via  any  combination  of  video,  voice,  data  and  Web  collaboration  -  on-demand.  Without  complex  IT  intervention 
or  advance  reservations.  A  single  dial-in  number  or  buddy  list  securely  connects  any  number  of  participants,  over  any  network, 
any  protocol,  any  speed  or  any  collaboration  device.  And,  you  can  do  it  all  with  confidence  since  Polycom  has  the  most  widely 
used  unified  solution  in  the  world.  It's  really  that  simple.  Isn't  it  time  you  demanded  on-demand  collaboration  from  Polycom? 

Get  a  free  copy  of  the  Unified  Collaborative  Communications  whitepaper  at  www.polycom.com/cio  or  call  1-877-POLYCOM. 


POLYCOM 


£7004  Polycom,  Inc.  All  rights  reserved.  Polycom  and  the  Polycom  logo  are  registered  trademarks  and  the  SoundStation  industrial  design  is  a 
trademark  of  Polycom,  Inc.  in  the  U.S.  and  various  countries.  All  other  trademarks  are  the  property  of  their  respective  owners. 
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Millions  of  your  peers  are  turning  to  the  Security  Guidance  Center  for  the  latest  in  security.  By  visiting  regularly, 
they  get  the  tools,  guidance,  and  training  needed  for  better  protection  against  viruses  and  other  security  threats. 
Visit  microsoft.com/security/IT  today  and  see  for  yourself  the  newest  additions,  including: 


Microsoft®  Windows®XP  Service  Pack  2  with  Advanced  Security  Technologies  Download  it  for  free 
and  evaluate  the  latest  updates  for  increased  system  control  and  proactive  protection  against  security  threats. 


Free  Online  Self  Assessment  Complete  this  free,  Web-based  self-assessment  test  to  help 
you  evaluate  your  organization's  security  practices,  and  identify  areas  for  improvement. 
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Go  today  to  microsoft.com/security/IT 
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No  matter  where  you  do  business. 


The  Brothei*  Advantage 

Comprehensive  selection 
>■  Increased  productivity 


>•  Lower  acquisition  costs 
Reduced  consumable  costs 
>■  24/7/365  support  and  service 
Free  evaluation  program 


Brother  Printer,  Fax  and  Multi-Function  Center®  models  — 
designed  to  increase  productivity  while  decreasing  overhead. 

Considering  that  over  94%  of  Fortune  1000  company  employees  work 
outside  corporate  headquarters*,  equipping  them  with  a  cost-effective 
solution  is,  to  say  the  least,  a  major  challenge. 

That's  why  Brother's  Commercial  Division  is  committed  to  providing 
superior  and  reliable  imaging  solutions  that  increase  productivity  while 
reducing  costs.  This  enables  businesses  like  yours  to  effectively  address  critical 
organizational  goals  and  challenges. 

But  it  is  our  product  reliability,  coupled  with  a  responsive  nationwide 
support  and  service  network,  that  has  companies  like  yours  putting  Brother  at 
the  top  of  their  requisition  lists. 


Desktop  Laser  Solutions 


« 

Network  Printer  Solutions 


Brother's  Commercial  Division  welcomes  the  opportunity  to  put  our 
resources  to  work  for  you.  Contact  us  today  so  we  can  show  you  how  we  can 
positively  impact  your  bottom  line  while  enhancing  your  performance. 

For  more  information,  call  1-866-455-7713. 


•Purchase  Influence  in  Larger  American  Businesses  (Erdos  &  Morgan,  2001). 
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Trend  Micro  and  Cisco  Systems —  working  together. 

Imagine  a  network  solution  so  advanced,  so  secure,  so  ingeniously  proactive, 

you  may  never  have  to  worry  about  an  outbreak  again. 

Find  out  more  at  www.trendmicro.com/cisco 


:  HOW  INDIA  BECAME  A  : 


OFFSHORE  OUTSOURCING 

India’s  popularity  as  an  out¬ 
sourcing  hotspot  didn’t  happen 
by  chance. 

Indian  professionals,  venture 
capitalists  and  entrepreneurs  of 
rigin  helped  promote  India 
tsourcing  destination, 
ng  to  a  study  conducted 
'orld  Bank  Institute  in 
on,  D.C.,  by  Evalueserve, 
ss  intelligence  and 
hfirm. 

While  other  low-cost  destina¬ 
tions  are  slowly  catching  up  with 
India  in  outsourcing,  the  subcon¬ 
tinent  will  retain  its  edge  because 
of  the  growing  influence  and 
expertise  of  the  Indian  diaspora, 
particularly  inthe  United  States, 
Canada  and  the  United  Kingdom, 
the  study  concluded.  A  key  factor 
in  favor  of  India  is  the  increase  in 


organized  networking  and  men¬ 
toring  that  the  members  of  the 
diaspora  community  can  provide 
to  businesses  engaged  in  out¬ 
sourcing,  according  to  the  study. 

By  the  1990s,  many  Indian 
engineers,  who  started  moving  to 
the  United  States  in  the  1960s, 
had  either  become  entrepreneurs, 
venture  capitalists  or  senior  exec¬ 
utives  in  large  and  midsize  com¬ 
panies,  according  to  the  study. 
Many  of  these  professionals 
started  their  own  firms  in  India, 
while  others  persuaded  their 
companies  to  hire  Indian  IT  pro¬ 
fessionals.  This  provided  more 
visibility  to  the  Indian  talent  pool 
and  resulted  in  the  strengthening 
of  the  diaspora.  For  example,  by 
late  1999,  Indians  constituted 
approximately  24  percent  of  the 
IT  professional  population  of  Sili- 


con  Valley,  according  to  the  study.  those  of  Indian  origin,  are  funding 
Some  venture  capitalists  in  companies  that  have  back-end 
the  United  States,  particularly  Continued  on  Page  28 
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AUTOMOTIVE  MANUFACTURING 

The  relationship  between  General 
Motors  and  the  software  industry  is 
bound  to  get  much  more  interesting 


over  the  next  few  years  as  software 
becomes  a  more  important  component  of  the  automobile. 
That’s  the  assertion  of  Anthony  Scott,  the  CTO  of  GM’s  Infor¬ 
mation  Systems  and  Services  organization,  who  spoke  at 
SoftSummit  2004,  a  software  industry  conference. 

The  amount  of  software  loaded  into  a  typical  car  is 
skyrocketing.  In  1990,  cars  contained  approximately 


1  million  lines  of  software  code;  by  2010,  Scott  predicted, 
cars  will  contain  100  million  lines.  That  means  that  a 
much  broader  range  of  software  will  be  used  in  tomor¬ 
row’s  cars.  Remote  diagnostics  software,  media  players 
and  even  database  software  will  run  on  automobiles, 
Scott  said. 

That  adds  up  to  new  challenges  for  GM  and  the 
software  developers  who  work  with  the  company.  Scott 
said  the  automaker  will  be  looking  to  the  software 
industry  to  assume  greater  responsibility  for  product 
liability  and  quality  assurance.  -Robert  McMillan 
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60,000  traders,  in  1,400  trading  offices  across  51  countries,  trust  us  to  ensure  that  their  information  gets 
to  the  right  person,  in  the  right  place,  at  the  right  time.  Our  unique  network  expertise  has  enabled  seamlessly 
integrated  applications  and  devices  that  span  the  globe.  Allowing  global  finance  to  flow  freely  around 
the  world.  Discover  more  about  succeeding  in  the  digital  networked  economy  at  bt.com/networkedlT 


More  power  to  you 


A  third  of  all  financial  traders 
use  BT’s  networked  IT  services 
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DONAT 
YOUR  PRINTER 
TO  SCIENCE 


MEDICAL  RESEARCH  Looking  for  somewhere  to  toss  your  old  ink-jet 
printers?  A  team  of  scientists  working  to  create  human  tissue  may  have  a 
good  use  for  them.  Ink-jets  that  are  10  years  old,  they  say,  are  perfectly 
suited  to  create  sheets  of  human  skin  and  other  tissue  that  one  day  may 
help  burn  victims  and  even  manufacture  organs. 

Vladimir  Mironov,  director  of  Shared  Tissue  Engineering  Laboratory  at 
the  Medical  University  of  South  Carolina,  is  one  of  the  scientists  who  has 
rigged  Hewlett-Packard  and  Canon  ink-jet  printers  to  shoot  out  proteins 
instead  of  ink,  and  to  capture  tissue  on  specialized  gel  instead  of  paper. 
Older  printers  work  well  because  their  spray  nozzles  have  larger  holes  and 
are  less  likely  to  damage  fragile  cells. 

The  “skin  printing”  research,  although  in  early  stages,  aims  to  replace 
the  current  skin-graft  method,  which  can  lead  to  postoperative  complica¬ 
tions,  says  Dr.  Anthony  Atala,  a  researcher  at  Wake  Forest  University  School 
of  Medicine.  Today,  burn  victims  receive  skin  grafts  from  unburned  parts 
of  their  body  or  from  skin  tissue  artificially  grown  in  a  lab.  But  trouble  can 
arise,  particularly  when  the  body  rejects  grafts  that  don’t  exactly  repli¬ 
cate  human  tissue.  Also,  grafted  skin  can  tighten  overtime,  causing 
discomfort  and  itching.  Skin  made  from  ink-jet  printers  may  come  closer 
to  replicating  human  tissue,  Atala  says,  because  it’s  created  using  skin- 
tissue  cells.  While  skin  printing  begins  with  the  same  process  of  cultivat¬ 
ing  cells  used  in  skin  grafting,  Atala  says  that  the  printers  create  skin 
more  efficiently.  “We’re  seeing  a  better  quality  skin  that  will  cover  more 
area,”  he  says.  “The  quality  of  the  tissue  is  higher." 

Thomas  Boland,  an  assistant  bioengineering  professor  at  Clemson 
University  and  another  researcher  involved  in  the  project,  says  he  came 
up  with  the  idea  one  day  when  overseeing  students  who  had  become 
frustrated  with  earlier  research  trying  to  "stamp"  skin  cells.  “I  went  to  the 
lab  to  look  around  and  saw  an  unused  ink-jet  printer  sitting  there  in  the 
lab.  I  thought,  Why  not  use  that?” 

Atala  and  Boland  say  the  technology  could  be  used  clinically  in  a  few 
years  for  burn  injuries,  accidents  and  extra  skin  coverage.  After  that,  the 
researchers  hope  they  can  create  other  types  of  organs  and  even  body 
parts  using  ink-jet  printers.  If  they  are  successful,  the  possibilities  are 
nearly  limitless:  printed  organs  could  be  created  for  use  in  transplants 
and  for  drug  testing:  and  the  technique  could  even  allow  plastic  surgeons 
to  reproduce  Nicole  Kidman’s  nose,  for  example  (if  she  were  to  donate 
her  cells). 

Sound  futuristic?  Those  types  of  cosmetic  and  transplant  applications 
probably  are,  says  Atala.  “As  you  get  into  more  complex  tissues,  you  need 
more  ingredients  and  we’re  still  working  on  that.”  -Susannah  Patton 
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Offshore  Outsourcing 

Continued  from  Page  26 

operations  in  India  to  save  on  R&D  costs,  the 
study  found.  As  of  March,  more  than  150  U.S.- 
based  startups  had  some  back-end  operations 
in  India;  by  March  2006,  the  study  estimated 
that  this  number  will  likely  double. 

Offshoring  R&D  is  a  key  strategy  of  startups 
because  fewer  funds  are  available  to  such  com¬ 
panies  now  than  before  the  dotcom  boom  and 
bust,  says  Promod  Haque,  managing  partner  of 
Norwest  Venture  Partners.  Whether  the  work  is 
outsourced  to  India  or  to  another  offshore  out¬ 
sourcing  location  can  to  an  extent  be  influenced 
by  where  key  employees  come  from,  says  Haque, 
who  is  Indian.  But  in  the  end,  such  decisions  are 
ultimately  made  on  business  grounds,  he  says. 

The  sudden  demand  for  skilled  labor  fostered 
by  both  the  Internet  boom  and  Y2K  would  have 
drawn  India’s  engineers  and  technicians  into  the 
global  IT  industry  regardless  of  the  diaspora’s 
role,  the  study  concluded.  It  just  so  happened 
that  the  diaspora  helped  jump-start  India’s  IT 
industry,  and  when  the  boom  occurred,  India 
was  already  a  viable  option  for  offshore  work. 

On  the  contrary,  countries  that  don’t  have  a 
similar  diaspora  of  expatriates  are  playing  catch¬ 
up.  “Countries  such  as  South  Africa,  Russia  and 
other  Eastern  European  countries  were  not  simi¬ 
larly  drafted  into  the  boom,  and  we  believe  that 
the  influence  of  the  diaspora  has  been  crucial,” 
says  Alok  Aggarwal,  a  cofounder  of  Evalueserve, 
which  has  operations  in  Chappaqua,  N.Y.,  and 
in  India. 

While  expatriates  agree  that  the  diaspora  has 
been  a  major  catalyst  for  India's  outsourcing 
boom,  many  downplay  its  role  in  the  gradual 
transformation  of  India  into  a  global  IT  force. 

Prakash  Gurbaxani,  CEO  of  TransWorks  Infor¬ 
mation  Services,  a  Mumbai-based  business 
process  outsourcing  company,  says  that  the  role 
of  the  Indian  diaspora  has  been  overrated.  The 
boom  in  offshoring  to  India  was  not  driven  by 
Indians  abroad,  but  by  companies  like  General 
Electric,  which  back  in  the  1990s  saw  value  in 
outsourcing  to  India,  Gurbaxani  says.  Indians  in 
an  organization  may  be  asked  to  help  facilitate 
an  offshore  move  because  of  their  knowledge  of 
the  country,  but  they  don’t  influence  the  initial 
decision  of  the  large  multinationals  to  out¬ 
source,  he  adds. 

The  decision  to  outsource  to  India  comes 
from  “India’s  reputation  as  a  location  for  low- 
cost  and  quality  services  that  companies  like  GE 
helped  build,”  Gurbaxani  says.  -John  Ribeiro 
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Managing  some  of  the  parts  or 
the  sum  of  the  parts? 


holistic:  // 

(whole4s'tic) 

the  importance  of 
the  whole  and 
interdependence 
of  its  parts. 


Maximize  IT  value  with  holistic  IT  management  and  governance 

IT  is  a  complex  business  within  a  business:  a  set  of  interdependent,  business-critical  functions  that 
your  organization  relies  on  to  succeed.  Effective  governance  requires  seamless  control  and  a  clear 
understanding  of  how  these  parts  work  together.  Compuware  IT  Governance  by  Changepoint  gives 
you  the  power  to  manage  IT  as  an  integrated  whole  and  maximize  the  value  of  every  project,  application 
and  infrastructure  investment. 


Awarded  “Best  Solution”  by  attendees  at  the 

Gartner  Project  and  Portfolio  Management  2004  Conference 


Visit  our  Governance  Resource  Center  at 

www.compuware.com/holistic 

for  expert  views  on  IT  Governance 
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High-Tech 


gadgets  As  the  holiday  season  quickly  descends  on  us, 
thoughts  naturally  turn  to  gift-giving.  For  techies  and 
nontechies  alike,  there  are  any  number  of  high-tech 
gadgets  that  are  useful  and  fun.  Here  are  just  a  few  that 
might  make  the  folks  on  your  gift-giving  list  happy. 

-Megan  Santosus 


WHY  NOT  STUFF  THE  STOCKING  OF  THE 
OUTDOORSY-TYPE  WITH  THE  SWISS  MEMORY, 

the  latest  multitooled  pocket  knife  from  Victori- 
nox,  maker  of  the  famous  Swiss  Army  Knife. 

In  addition  to  the  typical  blade,  scissors,  nail 
file/screwdriver  combo  and  key  ring,  the  Swiss 
Memory  features  a  64MB  USB  memory  stick 
that’s  compatible  with  several 
versions  of  Windows,  Mac  and 
Linux  operating  systems.  And  with 
available  Secure-LOCK  software, 
the  Swiss  Army  folks  are  looking 
out  to  keep  anyone’s  pocket-knife- 
based  data  safe.  For  more  informa¬ 
tion,  visit  www.swissarmy.com. 


FOR  A  GIFT  THAT’S  SLEEK  AND  STYLISH  AS  WELL  AS 
FUNCTIONAL,  CHECK  OUT  WORLDGATE  COMMUNICA¬ 
TIONS’  OJO  PERSONAL  VIDEO  PHONE.  The  Ojo  uses 
advanced  video  compression  technology,  high-speed  modems 
and  proprietary  video  enhancement  algorithms  to  produce 
what  WorldGate  describes  as  “true  to  life”  picture  quality.  The 
standalone  phone  provides  for  video  messaging,  picture- 
based  caller  ID  and  onscreen  directory  information,  and  can 
be  set  up  without  additional  household  wiring.  The  Ojo  may  be 
just  the  thing  for  the  person  on  your  list  who  wants  to  reach  out 
and  see  someone.  See  www.wgate.com  for  more  details. 


HAVE  MUSIC  FANS  ON  YOUR  LIST 
WHO  LOVE  MP3S?  THEN  OAKLEY 
MAY  HAVE  THE  PERFECT  GIFT.  The  eyewear  and 
apparel  maker  has  just  introduced  THUMP,  digital  audio 
sunglasses  complete  with  multidirectional  earphones 
that  allow  wearers  to  be  both  fashionable  and  tuned  in  to 
their  favorite  tunes.  The  1.8-ounce  glasses,  which  are 
available  in  seven  frame  colors  and  lens  combinations, 
come  in  two  versions;  the  128MB  model  holds  more  than 
two  hours  of  music  while  the  256MB  model  supports 
more  than  four  hours  of  listening  enjoyment.  With  a  high¬ 
speed  USB  connection,  Thump  enables  wearers  to 
quickly  download  all  their  favorite  music  and  take  it  on  the 
go.  Check  out  www.oakley.com  for  more  information. 


WATCHES  THAT  ONLY  TELL  THE  TIME  ARE  SO  PASSE. 
THIS  YEAR,  THE  SWATCH  GROUP’S  ENTRY  INTO  THE 
“SMART  WATCH”  CATEGORY  IS  PAPARAZZI,  a  wrist 
accessory  that  allows  wearers  to  access  personalized 
information  via  MSN  Direct.  Available  in  four  styles, 
Paparazzi  features  “Swatch  City,"  specialized  content 
that  gives  wearers  the  heads-up  on  happenings  at  local 
bars,  nightspots  and  events.  In  addition,  wearers  can 
receive  customized  news,  sports,  stock  quotes  and  weather 
information.  Paparazzi  is  waterproof  and  includes  a 
rechargeable  lithium  battery.  More  details  are  available 
at  www.swatch.com.  — 


PHOTOS  CLOCKWISE  FROM  TOP:  OAKLEY;  SWATCH;  WORLDGATE:  SWISSMEMORY/VICTORINOX 


DB2  is  middleware,  but  it  is 
anything  but  middle-of-the-road. 

In  fact,  DB2  is  part  of  an  innovative 
family  of  information  management 
products  that  can  integrate  and 
actually  add  insight  to  your  data. 
That’s  big. 

DB2  is  also  the  leading  database 
built  on  and  optimized  for 
Linux?  UNIX*"  and  Windows?  built 
to  take  full  advantage  of  your 
existing  heterogeneous  and  open 
environments,  and  built  to  enable 
true  grid  computing. 


Plus,  there’s  no  constricting  contract. 

DB2  is  also  middleware  with  an  eye  on 
your  resources.  All  of  them.  An  ITG 
study  showed  overall  costs  for  Oracle 
Database  are  up  to  four  times  higher 
than  DB21  A  Solitaire  study  found  that, 
on  average,  Oracle  Database  required 
25%  more  time  to  manage  than  DB2? 
And  the  Transaction  Processing 
Performance  Council  showed  DB2  as 
the  overall  price/performance  leader  for 
TPC-C  on  Linux,  UNIX  and  Windows. 
Ahead  of  both  Oracle  Database  and 
Microsoft®  SQL  Server1? 


Then  there’s  this:  Oracle  will  drop  the 
current  level  of  support  for  Oracle 
Database  8i  at  the  end  of  2004.  Meaning 
limited  support,  higher  cost  or  a 
complete  migration  to  current  versions. 
Fortunately,  IBM  offers  ongoing,  around- 
the-clock  service  and  support  for  DB2. 

Why  not  move  up  to  middleware  that 
makes  sense?  Through  the  end  of  the 
year,  you  can  get  IBM  DB2  Universal 
Database  by  taking  advantage  of 
our  extremely  compelling  trade-up 
promotion.  Visit  ibm.com/db2/swap 
today  to  find  out  if  you  qualify. 
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DEMAND  BUSINESS 
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IBM,  the  IBM  logo,  DB2  and  the  On  Demand  logo  are  trademarks  or  registered  trademarks  of  International  Business  Machines  Corporation  in  the  United  States  and  other  countries.  Linux  is  a 
registered  trademark  of  Linus  Torvalds.  Microsoft  and  Windows  are  registered  trademarks  of  Microsoft  Corporation  in  the  United  States  and/or  other  countries.  UNIX  is  a  registered  trademark 
of  The  Open  Group  in  the  United  States  and/or  other  countries.  Other  company,  product  and  service  names  may  be  trademarks  or  service  marks  of  others.  ©2004  IBM  Corporation.  All  rights 
reserved.  ’“IBM  Solutions  for  PeopleSoft  deployment  in  Mid-sized  businesses  Quantifying  the  New  Cost/Benefit  Equation,”  July  2003,  International  Technology  Group,  Los  Altos,  California. 
2“DB2  Performance  on  IBM  eServer*  pSeries  and  xSeries,"  Solitaire  Interglobal  Ltd.,  2003:  based  on  Oracle  Database  9i.  3AII  referenced  results  are  current  as  of  09/28/04.  Linux:  DB2  UDB 
v8.1 : 1.61  US$/tpmC,  18,661  tpmC,  available  12/15/04,  vs  Oracle  lOg:  3.94  US$/tpmC,  136,111  tpmC,  available  03/05/04.  UNIX:  DB2  UDB  v8.1 : 4.95  US$/tpmC,  809,144  tpmC,  available  09/30/04, 
vs  Oracle  lOg:  5.26  US$/tpmC,  371,044  tpmC,  available  09/30/04,  Windows:  DB2  UDB  v8.1: 1.68  US$/tpmC,  18,318  tpmC,  available  04/14/04,  vs  Microsoft  SQL  Server  2000: 1.85  US$/tpmC, 
22,052  tpmC,  available  02/18/04,  vs  Oracle  lOg  :4.98  US$/tpmC,  291,413  tpmC,  available  10/25/04.  TPC  Benchmark,  TPC-C  and  tpmC  are  trademarks  of  the  Transaction  Processing  Performance 
Council.  For  further  TPC-related  information,  please  visit  www.tpc.org 
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FORENSIC  EXPERT  JOHN  SGROMOLO 


Q&A  Devices  that  can  be  used  to  steal  data  are  not  limited  to  computers.  Camera  phones, 
flash  cards,  PDAs,  smartcards  and  other  devices  can  all  store  (and  therefore  be  used  to  steal) 
digital  data.  In  such  an  insecure  environment,  understanding  digital  forensics— the  science 
of  gathering  legal  evidence  in  digital  media— is  vital.  WHAT  DO  CIOs  NEED  TO  KNOW?  That’s 
what  contributor  Michael  Jackman  asked  John  Sgromolo,  a  former  special  agent  with  the 
United  States  Naval  Criminal  Investigative  Service  and  now  president  of  Digital  Forensics,  a 
consulting  and  investigative  company. 


CIO:  Why  is  digital  forensics  important  to  an  enterprise? 

JOHN  SGROMOLO:  In  addition  to  investigating  illegal 
activities,  digital  forensic  capabilities  are  necessary  to 
ensure  compliance  with  government  regulations  such 
as  Gramm-Leach-Bliley  and  Sarbanes-Oxley.  They’re 
also  necessary  for  enforcing  internal  policies,  such  as 
prohibiting  employees  from  browsing  pornographic 
sites  or  using  corporate  computers  for  personal  e-mail. 

Forensic  procedures  should  be  up  to  speed  in  advance 
of  any  litigation  or  enforcement  of  internal  regulations. 
Have  legal  counsel  brief  you  as  well  as  write  policies 
and  procedures  with  IT  and  security  staff. 

What  situations  might  require  forensic  analysis  of 
digital  devices? 

The  greatest  fear  for  enterprises  is  theft  of  proprietary 
data  or  trade  secrets. 

Unfortunately,  large-scale  theft  of  trade  secrets  has 
become  more  feasible  recently.  Thanks  to  the  introduc¬ 
tion  of  free,  Web-based  e-mail  accounts  with  gigabytes 
of  storage,  it  is  now  possible  for  a  data  pirate  to  simply 
e-mail  customer  lists  or  other  proprietary  data  to  a 
personal  account. 

Other  situations  include  harassment  (sexual  or  other¬ 
wise),  misuse  of  company  assets,  electronic  discovery 
requests  during  litigation,  and  fraud,  to  name  just  a  few. 

What  tools  are  available  to  conduct  digital  forensics? 

Software  used  by  forensic  investigators  ranges  from 
applications  as  simple  as  Notepad,  to  custom  propri¬ 
etary  software  that  costs  many  thousands  of  dollars. 
Some  open-source  DOS  or  Unix  utilities  are  available 
on  the  Internet  (see  www.opensourceforensics.org ). 

The  Forensic  Toolkit  by  AccessData  incorporates  DOS, 
Windows  and  Linux  tools  and  combines  capabilities 
for  acquiring  and  examining  evidence.  Other  products 
include  EnCase  Forensic  Edition  by  Guidance  Software 
and  WinHex  by  X-Ways  Software  Technology.  Special 
forensic  computers  are  designed  to  clone  or  copy  digital 
media  in  a  way  that  is  forensically  sound  and  can  meet 
a  court  challenge.  Whatever  software  and  hardware  you 
ultimately  choose,  the  key  to  using  them  effectively  is 
proper  training.  Vendors  frequently  offer  training. 
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Look  for  the  XML  Label 

data  standards  An  XML-based  format  for  pharmaceutical  label  data 
promises  to  make  medication  safer— but  not  until  late  next  year. 

The  Structured  Product  Labeling  (SPL)  format  will  help  the  Food  and 
Drug  Administration  ensure  uniform  application  of  product  labeling 
standards  for  prescription,  over-the-counter  and  veterinary  drugs, 
allowing  consumers  to  retrieve  drug  label  information  more  easily. 

“There  are  multiple  groups  who  will  benefit  from  the  use  of  this 
standard,”  says  Dr.  Randy  Levin,  director  for  health  and  regulatory  data 
standards  for  the  FDA.  The  FDA  itself  will  be  able  to  review  label  changes 
more  quickly  than  it  can  currently;  drug  companies  will  be  able  to 
exchange  label  information  more  efficiently;  and  health  information 
providers  will  have  access  to  the  data  in  a  computer-usable  format.  “All 
of  these  [capabilities]  are  geared  to  benefiting  the  users  of  the  labeling- 
including  the  prescriber,  dispenser  and  patient— by  promoting  patient 
safety  and  by  making  the  medication  information  more  accessible,” 

Levin  says.  The  SPL  format  is  part  of  the  FDA’s  and  the  National  Library 
of  Medicine’s  (NLM)  collaborative  DailyMed  initiative,  a  plan  that 
involves  storing  all  pharmaceutical  label  information  at  the  NLM  and 
providing  a  single,  easily  accessible  repository  for  the  data.  Health-care 
providers  could  then  retrieve  this  data  and  make  it  available  to  patients. 

DailyMed  was  originally  scheduled  for  deployment  this  past  June,  and 
pharmaceutical  companies  would  have  been  required  to  move  to  SPL  by 
that  date.  But  technical  delays  have  pushed  the  SPL  deployment  back  to 
October  2005.  This  is  good  news  for  the  pharmaceutical  companies 
involved,  as  there  was  originally  no  grace  period  planned  for  the  transi¬ 
tion,  says  Sarah  Powell,  director  of  product  strategy  for  First  Consulting 
Group,  which  sells  a  document  management  system  aimed  at  the 
pharmaceutical  industry.  “Most  of  the  companies  involved  have  their 
current  labeling  in  Microsoft  Word  rendered  to  PDF  for  submission,” 
Powell  says.  To  comply  with  DailyMed,  “Now  they  must  convert  Microsoft 
Word  to  XML  and  then  add  all  the  meta-data  associated  with  the  format.” 


-Christopher  Lindquist 
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“Citrix  helps  HP  increase 
productivity  by  giving  our 
employees  easy  and  secure 
access  to  standardized 
business-critical  applications  - 
from  anywhere  in  the  world.” 


dTRIX 


©2004  Citrix  Systems,  Inc.  All  rights  reserved.  Citrix  is  a 
registered  trademark  of  Citrix  Systems.  Inc.  in  the  U.S.  and 
other  countries.  All  other  trademarks  and  registered  trademarks 
are  the  property  of  their  respective  owners. 


Gilles  Bouchard,  CIO 
and  EVP  of  Global  Operations 

Hewlett-Packard  Company 


INFRASTRUCTURE  FOR  THE  ON-DEMAND  ENTERPRISE 


As  a  constantly  expanding  global  corporation, 

HP  faces  a  variety  of  IT  challenges  to  ensure  its 
145,000  employees  around  the  globe  have  access  to 
the  bandwidth-intensive  applications  they  need  to 
do  their  jobs.  HP’s  global  operations  require  access 
to  critical  applications  in  far-reaching  corners  of 
the  world  and  deployment  of  applications  across 
a  diversity  of  platforms.  So  HP  did  what  99%  of 
the  Fortune  500  have  already  done.  They  turned 
to  Citrix.  Now  HP  is  able  to  provide  its  employees 
with  secure  access  to  information  —  regardless  of 
location,  platform  or  device  used.  Citrix  is  helping 
HP  —  and  120,000  other  customers  —  save  money 
and  reduce  IT  complexity.  We  call  it  the  on-demand 
enterprise.  To  learn  what  Citrix  can  do  for  your 
business,  call  888-820-7918  or  visit  www.citrix.com. 
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Off  the  Shelf 


Edited  by  Carol  Zarrow 


Guides  to  the  Good  Life 


“In  being  slower,  time 
is  more  capacious. 
The  event  is  only  in 
the  moment.  By 
speeding  through 
life  with  technology, 
you  reduce  what  any 
given  moment  can 
hold.  By  slowing 
down,  you  expand  it.” 

-BETTER  OFF,  By  Eric  Brende 


“Ido,  in  fact,  place  a 
good  deal  of  stress  on 
the  obvious  in  this 
book,  and  that  is  quite 
deliberate.  In  logic,  as 
in  life,  it  is  the  obvious 
that  most  often  bears 
emphasizing,  because 
it  so  easily  escapes 
our  notice." 

-BEING  LOGICAL,  By  D.Q.  Mclnerny 

Life  is  full  of  surprises 
and  there  are  no 
guarantees,  but  one 
thing  is  certain: 

A  life  lived  without 
pleasure,  beauty,  and 
a  sensible  degree  of 
self-indulgence  is  a 
sad  and  wasted  one.” 

-THE  HEDONISM  HANDBOOK, 

By  Michael  Flocker 


With  the  holidays  here,  can  New  Year's  resolutions 
be  far  behind?  Simpler  living,  wiser  thinking  or  putting  the  fun 
back  into  life:  If  these  good  intentions  are  on  your  shortlist  for 
2005— or  if  you  know  someone  whose  shortlist  they  should  be 
on— here’s  a  how-to  trio  just  in  time  for  the  gift-giving  season. 


BETTER 


FLIPPING  i:-  SWITCH. ON  TECHNOLOGY  - 

K  ERIC  BRENDE 


M 


Better  Off:  Flipping  the  Switch  on  Technology 

By  Eric  Brende 

HarperCollins  Publishers,  2004,  $24.95 

I  ost  people  realize  that  we  take  electricity  for  granted,  and  that  in  those  rare  times  when  the  grid 
I  fails,  modern  life  grinds  to  a  halt.  But  what  about  the  rest  of  our  technologies— everything 
motorized  and  computerized— the  things  that  we  humans  created  to  serve  us,  yet  which  somehow 
seem  to  govern  our  lives?  As  part  of  his  graduate  studies  at  MIT,  Eric  Brende  aimed  to  find  out  how  it 
would  feel  to  do  without  all  technology  and  to  answer  the  questions,  How  much  is  enough?  What 
is  the  least  technology  we  need  to  achieve  the  most?  Brende  and  his  wife  spent  18  months  living 
in  what  the  author  calls  a  “Minimite”  community— think  Amish  with  even  fewer  luxuries— a  village 
boasting  a  harness  and  buggy  shop,  a  blacksmith  and  a  water  mill.  Brende  chronicles  his  often 
humorous  experiences  there  and  tells  us  what  many  may  already  suspect:  We  would  have  better 
health  and  peace  of  mind  if  we  shed  many  (though  not  quite  all)  of  the  technologies  that  so  define 
our  existence.  -Richard  Pastore 


Being  Logical:  A  Guide  to  Good  Thinking 

By  D.Q.  Mclnerny 
Random  House,  2004,  $19.95 


D.  Q.  Mclnerny 


BEING  LOGICAL 


These  days,  our  schedules  are  so  jam-packed  that  entertaining  a  calm,  well-reasoned  thought 
seems  like  an  unattainable  dream.  Being  Logical’s  brief  chapters  outline  simple  ways  to  recon¬ 
struct  your  outlook  and  remove  cluttering  details  from  your  thought  process.  In  order  to  work 
through  a  problem,  logical  thinking  is  a  must.  To  be  logical  is  to  be  human,  but  unfortunately,  human 
nature  creates  many  barriers  to  clear  thinking.  Vague  language,  quick  assumptions  and  cynicism 
make  it  hard  to  reach  a  logical  solution.  Mclnerny’s  focused  discussion  on  the  elements  of  a  produc¬ 
tive  argument  will  help  you  easily  resolve  problems  through  effective  communication.  And, 
with  suggestions  for  how  to  speak  clearly  and  think  optimistically,  your  thoughts  will  start  feeling 
clearer  in  no  time.  -Margaret  Locher 


MICHAEL  FLOCKER  THE  II  E  DO  MS  M  IIAMHIOOK  MASTERING  THE  LOST  AITS  IF  IEIS0IE  AND  PLEASURE 


The  Hedonism  Handbook:  Mastering  the 
Lost  Arts  of  Leisure  and  Pleasure 

By  Michael  Flocker  _ 

Da  Capo  Press,  2004,  $12.95  “ 

Americans,  in  their  misguided  pursuit  of  the  good  life  through  60-hour  workweeks,  low-carb  diets 
and  intense  workouts,  are  making  themselves  miserable.  What  the  overworked  and  uber- 
regimented  need,  says  the  author  in  this  irreverent  paperback,  is  a  stiff  drink  and  a  roll  in  the  hay. 
But  seriously,  folks— even  if  this  book  is  not— Flocker  reclaims  Epicurus,  the  Greek  philosopher  who 
maintained  that  pleasure  ( hedone )  is  the  highest  good  in  life,  for  the  modern  age.  With  breezy  bios 
of  history’s  most  famous  hedonists,  copious  quotes  on  the  importance  of  having  fun,  and  mantras 
and  lists  galore,  Flocker  offers  tested  tips  on  slowing  down  our  lives  and  sensible  advice  for  working 
smarter  (not  harder).  His  book  will  elicit  a  chuckle  if  nothing  else.  -Meridith  Levinson 
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Were  Resources  Connection. 
We  create  value  for  clients  by 
helping  them  execute  their 
strategies  more  cost-effectively. 
We  began  as  part  of  a  Big  Four 
firm;  now  we  are  independent 
and  publicly  traded.  Our 
heritage  attracts  the  best  project 
specialists,  veterans  of  the  Big 
Four  firms  and  FORTUNE  500® 
companies  —  so  they  know  how 
it  feels  to  give  that  nod. 


800-900-1131 

resourcesconnection.com 


IT’S  ONLY  AN  INCH  OF  MOVEMENT, 

BUT  A  NOD  IS  HARD  TO  DO. 

It’s  a  commitment:  to  a  project,  to  a  mission, 
to  the  direction  of  a  company. 

With  our  Associates  on  your  project,  with  their  skills  and  experience, 
You  feel  freer  to  make  those  commitments. 

You  feel  freer  to  move  that  inch. 


*  RESOURCES 

CONNECTION 


Get  there  with  people  who  have  been  there  before. 

Finance  and  Accounting,  HR,  IT,  Internal  Audit  and  Supply  Chain 
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On  the  Move 


By  Meridith  Levinson 


A  Fateful  Call 


PROMPTS  DAWN  LEPORE 
TO  SHIFT  GEARS 


DAWN  LEPORE  WAS  ON  VACATION  WITH 
HER  FAMILY  IN  JULY  2004  WHEN  THE 
THEN-VICE  CHAIRWOMAN  AND  CIO  OF 
CHARLES  SCHWAB  RECEIVED  A  CALL 
FROM  A  FRIEND  WITH  RECRUITING  FIRM 
SPENCER  STUART.  The  friend  had  a  tip 
about  the  top  position  with  Drugstore.com, 
an  online  retailer  that  targets  busy  profes¬ 
sional  women— just  like  Lepore— who  don’t 
have  time  to  hit  stores  for  shampoo  and 
other  essentials.  Kal  Raman,  president  and 
CEO,  resigned  from  the  company  the 
previous  month— the  same  day  the  com¬ 
pany  lowered  its  quarterly  and  yearly 
earnings  estimates,  and  Drugstore. corn’s 
board  was  looking  for  a  new  leader.  That 
Lepore  exemplified  Drugstore. corn’s  core 
customer  (on  top  of  her  experience  on  boards 
of  other  companies)  made  her  an  extremely 
attractive  candidate.  Lepore  was  intrigued 
by  the  same  thing;  indeed,  the  similarities 
between  herself  and  Drugstore. corn’s 
target  customers  played  a  significant  role 
in  her  decision  to  join  the  company  as 
chairwoman  and  CEO  this  October. 

Lepore’s  phone  had  rung  off  the  hook 
with  calls  from  recruiters  looking  to  place 
her  during  the  Internet  boom.  But  atthattime, 


she  didn’t  want  to  leave  Schwab. 

The  company  was  doing  well 
and  Lepore  enjoyed  running  an 
IT  shop  that  was  on  the  cutting 
edge  of  technology.  She  was 
rewarded  for  her  work  with  an 
appointment  as  a  vice  chairwoman  in  2001, 
just  as  the  bear  market  led  Schwab's 
fortunes  south.  Three  years  into  her  tenure 
as  vice  chairwoman  of  technology  and 
administration,  she  was  primed  to  make  a 
move.  And  then  came  the  call  from  her 
friend  at  Spencer  Stuart. 

The  timing  appeared  fortuitous  to  Lepore 
for  a  variety  of  reasons.  For  one,  she  had 
been  mulling  the  next  phase  of  her  career, 
which  she  hoped  would  include  a  stint  as 
CEO.  The  call  also  happened  to  coincide 
with  four  years  of  downsizing  at  Schwab 
and  came  just  days  before  David  Pottruck's 
highly  publicized  departure  as  CEO. 

Lepore  pointedly  says  that  Schwab’s 
financial  challenges  and  the  management 
change  were  not  factors  in  her  decision 
to  join  Drugstore.com.  (After  all,  Drug¬ 
store. com  presents  financial  challenges— 
namely,  how  to  turn  the  company’s  growth 
into  profits.)  She  says  that  as  she  started 


to  get  more  calls 
from  recruiters 
over  the  past  year, 
she  began  to 
consider— given 
her  age (50) and 
the  course  of  her  career— the  possibility  of 
becoming  a  CEO  if  the  right  opportunity 
came  along. 

The  factors  that  did  play  a  role  in  Lepore’s 
decision  to  take  the  job  with  Drugstore.com 
were  her  passion  for  and  years  of  experi¬ 
ence  with  consumer-focused  and  Internet- 
based  businesses,  her  readiness  to  relocate 
from  San  Francisco  to  Seattle,  the  positive 
feedback  about  the  company  she  received 
from  female  friends  who  are  Drugstore.com 
customers,  and  the  sense  of  ownership  she 
gets  from  leading  a  company  for  which 
she’s  also  the  target  customer.  Says  Lepore, 
“The  majority  of  our  customers  are  women, 
and  many  of  them  have  similar  characteris¬ 
tics  as  I  do:  They  have  children.  They  have 
jobs."  They’re  so  busy,  she  adds,  that  it's 
11:00  p.m.  before  they  can  order  their 
children's  diapers  or  their  prescriptions. 

Now  that’s  a  CEO  who  knows  her  core 
customer. 


Three  weeks  after 
W.  ROY  DUNBAR 
joined  MASTER¬ 
CARD  INTERNATIONAL 
as  president  of  its  global 
technology  and  operations 
unit,  EDS  named  him  to 
its  board  of  directors. 
Dunbar’s  appointment 


toEDS’s  board  appears 
to  be  part  of  the  IT  services 
company’s  seemingly 
ongoing  effort  to  recruit 
current  and  former  CIOs— 
whether  on  its  board  or  in 
management  positions. 
Former  CIOs  Charlie  Feld 
and  Dave  Clementz  are 


also  with  EDS. 

After  being  hired  as  CTO 
of  Align  Technology,  a 
maker  of  teeth  straightening 
technology,  Cecilia  Claudio 
was  appointed  to  outsourc¬ 
ing  consultancy  Ram- 
pRate’s  board  of  advisers. 
Claudio  was  formerly  CIO  of 


Farmers  Insurance  and 
Zurich  Financial. 

Stephen  Yates,  the  for¬ 
mer  president  of  USAA’s  IT 
unit  who  left  the  company 
amidst  an  offshore  out¬ 
sourcing  scandal  in  May 
2004,  landed  at  KeyCorp 
as  its  executive  VP  and  CIO. 
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OUR  KNOWLEDGE  OF 
WHAT  CIOs  NEED  GOES  BACK 

-  and  forward  - 


A  LONG  WAY. 


While  the  title  of  CIO  may  not  have  existed  65  years 
ago,  that’s  how  long  Fujitsu  has  been  developing 
innovative  technology  solutions  for  the  world’s 
leading  companies. 

Today,  we’re  leveraging  our  multi-billion  dollar 
annual  R&D  budget  to  give  CIOs  the  powerful  new 
enterprise  solutions  they  need  to  automate  even 
the  most  complex  business  infrastructures.  Our 
customers  and  business  units  work  closely  with  our 
research  teams  to  guide  new  product  development. 
This  teamwork  ensures  our  discoveries  make  it  out  of 
the  lab  and  into  the  enterprise. 

Of  course,  there’s  a  lot  more  to  a  $45-billion 
company  than  just  R&D.  Our  high-performance  mobile 
computers,  scalable/reliable  servers,  and  managed  and 
professional  services  also  give  CIOs  all  the  tools  they 
need  for  a  world-class  IT  partnership.  For  details  go  to 
us.fujitsu.com/computers  or  call  I  -800-83 1  -3 1 83. 
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It’s  already  here. 


Captaris  is  Business  information  Delivery 


A  Captaris  solution  is  probably  somewhere  in  your  organization. 
It’s  in  data,  systems  and  workflows,  already  doing  its  job, 
so  embedded  that  it's  easy  to  overlook. 

It  helps  existing,  disparate  technologies  talk  to  each  other 
so  content  is  accessible  from  any  source,  in  any  format. 


It  automates  workflow  processes,  freeing  your  talent  to  focus 
on  truly  important  tasks.  It  delivers  information  to  any  channel 
so  it  can  be  used  anytime  to  add  the  ultimate  value  to 
your  organization. 

That’s  Business  Information  Delivery.  That's  Captaris. 
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Captaris  RightFax— Send,  receive  and  manage  enterprise  faxes  and 
e-documents. 

Captaris  Interchange-Generate  and  distribute  high-volume,  individualized 
documents  independent  of  source  data,  format  or  delivery  method. 

Captaris  Teamplate-Automate  business  processes  with  easy, 
flexible  and  integrated  workflow  solutions. 

Our  products  are  ready  to  be  deployed  individually  or  as  a  whole  to  extend  your 
IT  investments  and  help  you  become  a  more  innovative  and  agile  global  competitor. 
Uncover  more  about  Business  Information  Delivery  at  captaris.com/BID. 


^Captaris 

The  Way  Information  Moves 


Keynote 

New  Ideas  from  Leading  Lights 


The 

Dehumanized 

Employee 

IT’s  dark  side  lies  in  its  tendency  to  encourage 
bosses  to  see  people  as  bits  and  bytes. 

BY  JAMES  HOOPES 

IN  THE  1936  FILM  Modem  Times,  Charlie  Chaplin  is  fed  by  an  assem¬ 
bly  line  and  almost  eaten  by  one.  The  idea  that  machines  con¬ 
sumed  people  was  a  common  complaint  in  the  age  of  heavy 
industry.  Human  beings,  went  the  popular  critique,  had 
become  the  slaves  of  machines,  and  in  the  case  of  Chaplin’s 
character,  twitching  automatons.  Chaplin’s  movie  reflected  the 
factory  world  created  by  Frederick  W.  Taylor  (1856-1915),  the 
most  influential  management  guru  of  the  machine  age.  Taylor 
transformed  factories  from  fairly  plodding  places  into  models 
of  economic  rationality  where  capital  intensive  machinery  ran 
full  and  fast  in  order  to  maximize  return  on  investment.  Since 
the  machines  had  to  be  driven  hard,  so  did  the  people  who  ran 
them.  Taylor  used  the  stopwatch  and  piece-rate  pay  to  squeeze 
the  last  ounce  of  effort  out  of  workers.  Despite  his  tyrannical 
methods,  Taylor  managed  to  sell  himself,  at  least  for  a  time,  as 
a  prophet  of  democracy,  out  to  enrich  the  working  class  rather 
than  enslave  them.  Chaplin  disagreed. 

History  seems  to  be  repeating  itself  in  the  information  age. 
Today’s  management  gurus  have  long  promised  that  information 
technology  would  deliver  a  new  birth  of  freedom  for  corporate 
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employees.  Contemporary  prophets  of  management  often  pre¬ 
dict  that  information  technology  will  flatten  hierarchies, 
empower  employees  and  shrink  giant  companies  into  human 
communities.  In  his  recent  book,  The  Future  of  Work,  MIT’s 
Thomas  Malone  argues  that  IT  is  pushing  out  authority  to  the 
fringes  of  the  organization  and  empowering  workers  to  have 
more  autonomy  and  decision-making  power.  To  a  significant 
degree  those  prophecies  are  being  fulfilled,  especially  for  well- 
educated  employees  with  knowledge  and  skills  in  high  demand. 

Reducing  Employees  to  Bits  and  Bytes 

It  is  increasingly  clear,  however,  that  for  many  other  employees, 
the  management  gurus  prophesying  that  IT  will  democratize 
today’s  workplaces  are  just  as  wrong  as  Taylor  was.  IT  can  be 
used  not  only  to  liberate  human  beings  but  to  control  them.  In 
industries  employing  unskilled  workers,  such  as  retailing,  IT 


ILLUSTRATION  BY  ROGER  CHOUINARD 


THE  LEADER  IN  CUSTOMER 
ANALYTIC  APPLICATIONS  BRINGS 
YOU  A  NEW  GENERATION 
OF  ENTERPRISE  Bl  SOLUTIONS. 
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Cutthroat  competition.  Intense  profit  pressure.  Price  erosion.  Declining  loyalty.  Business  today 
dictates  working  not  just  harder,  but  smarter.  Meeting  these  new  business  demands,  and 
overcoming  the  inherent  challenges,  requires  a  new  generation  of  Bl  solutions  unavailable  from 
traditional  Bl  vendors. 

Siebel,  the  leader  in  customer  analytic  applications,  brings  you  Siebel  Business  Analytics -Bl 
solutions  that  empower  everyone  with  actionable  real-time  insight  from  the  largest  data 
warehouses  and  across  enterprise  sources.  With  a  mission-critical  architecture  that  supports 
multi-terabytes  of  data,  thousands  of  users,  and  24x7  availability.  And  proven  application 
solutions,  with  built-in  industry-specific  best  practices  that  are  flexible,  quickly  implemented, 
integrate  with  your  infrastructure,  and  provide  low  TCO. 

Our  innovative  Bl  solutions  have  catapulted  Siebel  into  the  leadership  position  in  analytic 
applications  for  sales,  marketing,  service,  and  senior  management,  with  over  75%  of  our  customers 
using  Siebel  Business  Analytics  beyond  CRM  and  across  a  full  range  of  enterprise  sources. 

Learn  why  the  most  analytically  sophisticated  companies  rely  on  Siebel.  Visit  www.siebel.com/newgen 


siebel 

Business  Analytics 

CUSTOMER  INSIGHT  •  ENTERPRISE  INTELLIGENCE 
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has  created  top-down  control  to  a  degree  that  Taylor  could 
only  have  dreamed  of.  Instead  of  working  for  small  organiza¬ 
tions,  many  retail  employees  work  for  giant  firms  that  dwarf 
even  the  behemoths  of  the  industrial  era. 

Management  by  Remote 

Just  as  the  machine  age  made  Chaplin’s  character  a  faceless 
automaton,  the  information  age  can  make  employees  invisi¬ 
ble  to  managers  who  may  be  tempted  to  think  of  them  as 
mere  bits  of  data.  Thanks  to  IT  and  the  instantaneous  global 
reach  of  today’s  large  companies,  many  of  the  management 
decisions  affecting  employees  are  made  from  a  great  geo- 

IT  is  enabling  retail  chains  to  schedule 
employees  from  a  distance  and  soon  will 
permit  managers  to  monitor  employees 
with  RFID  tags. 

graphical  and  organizational  distance,  which  offers  managers 
the  temptation  to  ignore  the  human  beings  affected  by  their 
decisions.  As  long  ago  as  1995,  Thomas  Davenport,  my  col¬ 
league  at  Babson  College,  warned  that  management  gurus 
were  in  danger  of  treating  employees  “as  if  they  were  just  so 
many  bits  and  bytes.” 

Where  workers  in  Taylor’s  time  at  least  knew  the  managers 
who  profoundly  affected  their  lives,  employees  in  large  organ¬ 
izations  today  often  have  no  human  connection  to  managers 
who  exert  enormous  control  over  them.  A  century  ago  it  was 
unusual  for  corporate  headquarters  to  make  a  decision  that 
affected  employees’  daily  lives.  It  took  a  big  decision  with  big 
consequences — the  decision,  say,  to  open  or  close  a  factory. 
Workers’  routine  operations  within  the  factory  were  still  con¬ 
trolled  by  immediate,  highly  visible  supervisors. 

Using  IT  to  Squeeze  Work  Hours 

But  in  our  time,  employees  are  often  affected  not  just  by 
epochal  decisions  in  the  life  of  a  company  but  by  routine  daily 
decisions  made  at  headquarters  located  in  another  state  or 
even  another  country.  At  Wal-Mart  and  other  large  retailers, 
for  example,  the  pressure  under  which  employees  work 
depends  not  just  on  their  immediate  supervisors  but  on  cen¬ 
tralized  decisions  emanating  from  computerized  headquarters 
that  tell  local  managers  how  many  (or  rather,  how  few)  person 
hours  they  are  expected  to  use  that  week. 

Even  at  the  level  of  what  in  Taylor’s  time  would  have  been 
called  the  “shop  floor,”  IT  can  separate  frontline  supervisors 
from  employees  and  make  it  easier  to  manage  harshly.  The  bur¬ 


geoning  call  center  industry — that  does  much  of  today’s  mar¬ 
keting,  sales  and  customer  service — relies  on  IT  systems  that 
often  end  up  subjecting  employees  to  high  levels  of  stress.  As 
employees  speak  with  customers  they  are  supervised  remotely  via 
technology  that  monitors  their  pace,  productivity  and  faithful¬ 
ness  to  the  software-driven  script  that  plays  across  the  com¬ 
puter  screen  in  front  of  them.  The  script  provides  precise 
instructions  for  how  call  center  agents  should  handle  each  call 
from  beginning  to  end.  At  many  organizations,  as  Simon  Head 
notes  in  his  book,  The  New  Ruthless  Economy,  a  call  center 
employee’s  only  human  contact  with  a  manager  may  occur  dur¬ 
ing  a  work  break  when  the  underling  is  rebuked  for  some  short¬ 
coming  identified  in  real-time  on  the  manager’s  computer  screen. 

There  are,  of  course,  companies  such  as  Amazon,  Ameri¬ 
can  Express,  eBay  and  Southwest  Airlines  that  give  their  front¬ 
line  employees  a  great  deal  of  autonomy  and  respect  their 
judgment  and  ability  to  provide  the  kind  of  service  that  keeps 
customers  loyal. 

Even  so,  technological  developments  now  on  the  horizon  seem 
likely  to  increase,  not  decrease,  restrictive  top-down  manage¬ 
ment.  New  sensor  technology  such  as  radio  frequency  identifi¬ 
cation  (RFID)  devices  will  inexpensively  monitor  truck  drivers, 
salespeople,  field  service  technicians  and  other  offsite  personnel 
who  have  traditionally  worked  out  of  their  bosses’  sight.  As  such 
employees  are  monitored  with  increasing  intensity  by  new  tech¬ 
nology,  the  amount  of  “management  by  stress”  in  American 
workplaces  seems  likely  to  rise. 

It  is  customary  to  blame  the  declining  prospects  of  American 
workers  on  the  fierce  competition  of  the  global  economy.  Com¬ 
panies,  it  is  said,  have  to  use  part-time  and  temporary  employ¬ 
ees,  slash  benefits,  hold  down  wages,  and  demand  more  effort 
in  order  to  keep  jobs  in  America  while  competing  with  employ¬ 
ers  of  cheap  labor  overseas.  Whatever  legitimacy  there  is  to 
that  argument,  it  is  not  the  whole  story  behind  the  grim  new 
reality  of  a  shrinking  middle  class  and  the  country  increasingly 
divided  into  a  “two-tier  America.”  Modem  management’s  mas¬ 
terful  use  of  high  technology  should  not  be  let  off  the  hook  for 
the  increasing  numbers  of  citizens  who  are  working  harder  and 
earning  less.  If,  as  some  assert,  there  is  a  global  “race  to  the  bot¬ 
tom,”  IT  is  helping  to  facilitate  that  race  by  enabling  retail 
chains  to  schedule  employees  from  a  distance,  without  regard 
for  what’s  going  on  in  those  employees’  lives,  by  allowing  call 
centers  to  squeeze  productivity  out  of  customer  service  repre¬ 
sentatives,  and  soon,  by  permitting  managers  to  monitor  offsite 
employees  with  RFID  tags. 

One  Solution:  Employee  Hotlines 

What  can  IT  managers  do  to  help  alleviate  such  restrictive  envi¬ 
ronments?  One  possibility  is  to  watch  for  ways  to  make 
employees  more  visible  to  management.  To  the  degree  that 
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E  v  e  r  y  t  h  i  n  g  . 

Black  Box  has  the  most  comprehensive 
product  line: 

•  Cables  &  Connectors 

•  Cabinets  &  Racks 

•  Testers  &  Tools 

•  Power  &  Surge  Protection 

•  Video  &  Multimedia 

•  Switches 

•  ServSwitch 

•  Printer  Devices 

•  Converters 

•  Line  Drivers 

•  Desktop  Telephony 

•  Modems,  CSU/DSUs,  &  Muxes 

•  Networking 

•  Custom  Solutions  and  much  more 

Black  Box  can  help  you: 

•  Design 

•  Install 

•  Maintain 

your  wired  and  wireless  network 
infrastructures, 

As  the  world's  largest  infrastructure 
provider,  Black  Box  can  design  and 
install  data  and  voice  systems  for  you- 
and  perform  MAC  work  and  ongoing 
maintenance  to  keep  your  networks 
connected. 

And  with  more  than  90,000  connectivity 
products  available,  we  can  help  you 
to  decide  what's  best  for  your  network. 

Our  FREE  24/7/365  hotline  Tech  Support 
answers  your  call  in  20  seconds  or  less, 
any  time  of  the  day! 

Everything's  in  the  Box! 


Operating  in  141  countries  around  the  world,  we're  your 
■CIS  jK  one  source  for  data  and  voice  products  and  services — 

backed  by  the  industry's  best  Tech  Support.  We  design, 
NETWORK  SERVICES  install,  and  maintain  wired,  wireless,  and  hybrid  networks. 


blackbox.com  •  877-ASK-BBOX  •  For  25%  off  coupon,  click  on  the  e-card  link  at  blackbox.com 
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Fortoo  long  now,  the  name  of  the  game  has  been  survival.  Cut  your  budget. 
Slash  your  staff.  Do  more  with  less,  or  just  do  less.  Postpone  it,  reduce  it  or 
outsource  it.  Think  small. 


But  thinking  small  is  for  losers. 


We’re  looking  for  organizations  and  leaders  playing  to  win,  not  just 
survive,  despite  business  conditions  that  continue  to  be  difficult 
and  restrictive, 

Boldness  requires  the  vision  to  see  where  your  business  could  go, 
where  IT  could  lead  it  and  then  investing— money,  time,  people  and 
brainpower— to  make  that  happen,  it  means  finding  new  ways  for  tech¬ 
nology  to  make  the  enterprise  more  profitable.  It  means  going  after 
new  customers,  in  new  markets,  with  IT  hel  ping  to  create  new  products 
and  systems  in  that  pursuit.  Bold  companies  look  for  imaginative  ways 
to  organize  their  resources,  their  staff  and  their  governance  to  enhance 
their  future  competitiveness. 

Boldness  means  embracing  significant  risk  for  the  sake  of  great 
reward. 

If  you  can  show  measurable  results  of  how  IT  has  enabled  and  led  bold 
initiatives  in  your  organization,  then  our  readers— your  peers— want  to 
know  about  you. 
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V. 

Be  recognized  as  one  of  the  Bold  100. 

Apply  now  for  the  18th  Annual  CIO  100  Awards. 


CIO  100  honorees  will  be 
honored  at  the  annual  CIO  100 
Symposium  &  Awards  Cere' 
mony  Aug,  21-23,  2005.  at  the 
Hotel  del  Coronado,  California. 
Honorees— and  their  bold 
ideas— will  also  be  featured  in 
the  Aug.  15,  2005,  issue  of  C/O, 

Learn  more  about  the  CIO  100 
and  get  an  application  on  our 
website.  Applications  available 
online  at  www.CIO.com/ciolOO. 


The  2005  CIO  100 
Awards  Ceremony  is 
proudly  underwritten  by 


CIO 


The  Resource  for 
Information  Executives 


Keynote 


\ 


these  conditions  result  from  invisibility  and  ignorance  rather 
than  cruelty  and  callousness,  making  IT  an  instrument  of  gen¬ 
uine  bottom-up  communication  would  be  a  help.  Perhaps 
something  can  be  learned  from  the  anonymous  employee  hot¬ 
lines  that  companies  such  as  Shell,  MCI  and  Pricewaterhouse- 
Coopers  have  adopted  and  many  others  are  putting  in  place  to 
catch  potential  ethical  and  legal  violations  in  our  post-Enron 
era.  IT  sounding  boards  that  let  employees  “sound  off”  anony¬ 
mously  and  at  will  to  an  ombudsman  might  give  management 
a  different  and  more  accurate  understanding  of  organizational 
culture  and  morale  than  periodic  surveys  administered  by 
hired  consultants. 

An  Employee  Revolution? 

But  of  course  no  form  of  bottom-up  communication  can  reduce 
workplace  harshness  in  companies  more  interested  in  talking 
the  talk  than  in  walking  the  walk  toward  a  more  just  and  gen¬ 
erous  society.  It  may  be  that  for  some  years  yet,  information 
technology  and  the  global  economy  it  supports  will  make  cor¬ 
porate  America  a  little  bit  more  like  the  world’s  least  envied 
societies,  where  the  many  are  dominated  by  the  few. 


Yet  managers  who  are  eager  to  use  IT  not  just  to  run  their 
businesses  more  efficiently  but  to  create  a  better  society  may  be 
wiser  in  the  long  run.  America  is,  after  all,  a  democratic  coun¬ 
try,  and  citizens  will  not  remain  passive  forever  if  increasing 
numbers  of  them  are  reduced  to  second-tier  stams.  When  Chap¬ 
lin  filmed  his  critique  of  the  machine  age  in  1936,  Congress 
had  just  passed  the  National  Labor  Relations  Act,  which 
assisted  in  the  formation  of  the  great  industrial  unions  that 
were  so  vital  a  part  of  mid-20th-century  American  prosperity. 
The  political  pendulum  has  swung  a  long  way  in  the  other 
direction  since  then,  but  nothing  is  forever.  The  pendulum  may 
change  directions  once  more  unless  managers  and  companies 
can  use  IT  not  to  continue  but  to  reverse  our  present  course 
toward  a  corporate  society  of  rich  and  poor,  powerful  and 
powerless,  who  are  ever  less  visible  to  each  other.  QZ3 


James  Hoopes  is  Murata  Professor  of  Business  Ethics  at 
Babson  College  in  Wellesley,  Mass.  His  most  recent 
book  is  False  Prophets:  The  Gurus  Who  Created  Modern 
Management  and  Why  Their  Ideas  Are  Bad  for  Business 
Today.  Send  feedback  to  letters@cio.com. 


We  All  Need  a  Little 

Help  Sometimes... 


Although  there  is  no  magic  pill  that  makes  choosing  a  telecommunications 
company  easy,  Thompson  Advisory  Group  can  be  the  secret  to  your 
success.  We  partner  with  companies  to  find  the  best  solution  to  your 
communication  needs.  Our  advisors  are  experts,  who  put  their  industry 
knowledge  to  work  for  you  so  you  won’t  waste  valuable  time  or  money. 
Let  us  be  your  resource  of  experience,  knowledge,  and  expertise  to  give 
you  the  competitive  edge  your  company  needs. 
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Call  us  today  at  8 1 7-25 1  -008 1 
for  a  complimentary  benchmark 
or  visit  www.i-tag.net 
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Field-Tested  Ideas  from  CIOs  for  CIOs 


IT  Buyer 
Beware 

Nothing  is  ever  as  easy  as  your  vendor  wants  you  to 
believe.  NCCI  CIO  Jeff  Fields  suggests  you  think 
twice  before  you  sign  on  the  dotted  line. 

BY  JEFF  FIELDS 

LAST  YEAR,  the  company  I  work  for  felt  that  new  advances  in  technology 
could  be  incorporated  to  more  efficiently  manage  and  update  all 
of  the  workers’  compensation  information  and  manuals  that 
our  customers  need  to  conduct  their  businesses.  NCCI  is  the 
nation’s  most  comprehensive  provider  of  workers’  compensa¬ 
tion  information  and  services,  and  our  customers  consist  of  insur¬ 
ance  carriers,  regulators  and  government  insurance  departments. 
They  depend  on  the  manuals  we  produce,  which  contain  rules, 
classifications,  state-specific  rates  and  more.  In  addition,  these 
manuals  also  offer  the  extensive  cross-references  and  keyword 
searches  that  our  customers  need  to  operate  their  businesses. 

With  our  existing  system,  it  was  difficult  for  customers  to 
find  the  required  manuals  quickly.  In  addition,  the  process  of 
revising  these  manuals  was  labor  intensive;  updating  the  dif¬ 
ferent  versions  of  manuals  involved  many  steps.  And  the  man¬ 
uals  needed  approval  across  different  divisions  before  they 
were  published,  which  also  slowed  down  the  process.  There 
was  a  lot  of  back  and  forth  communication  between  the 
authors  of  the  manuals  and  the  publishing  group,  and  changes 
were  hard  to  track. 


To  improve  this  process,  we  decided  to  employ  an  enter¬ 
prise  software  system  to  more  efficiently  manage  all  of  this 
information.  The  system  would  also  be  used  to  track  the 
regulatory  rate  filings  for  all  the  states  in  which  NCCI  does 
business.  We  thought  the  selection  process  would  be  quick  and 
easy.  After  all,  we  knew  what  outcome  we  wanted  and  just 
needed  to  find  the  right  software  at  the  right  price.  We  figured 
the  vendors  could  be  trusted  to  give  us  the  straight  scoop  on 
their  products  and  services. 

Boy,  were  we  wrong. 

The  Big  Pitch 

The  first  thing  we  did  was  set  up  a  cross-functional  team 
consisting  of  representatives  from  business,  IT  and  the  project 
management  office  to  develop  an  RFP.  Based  on  responses  to 
the  RFP,  two  software  vendors  were  selected  to  demonstrate 
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their  products.  The  vendors  did  a  good  job  of  targeting  the 
business  with  the  functionality  that  their  systems  had  to  offer. 
Their  presentations  sparked  a  lot  of  enthusiasm,  and  the  team 
was  eager  to  get  started  on  the  project. 

At  this  point,  I  was  asked  to  lead  a  comprehensive  evaluation 
from  a  strictly  IT  perspective.  We  concluded  that  one  vendor 
was  best  suited  to  do  the  workflow  and  content  management 
aspect;  the  other  one  was  primed  to  edit  the  electronic  manu¬ 
als  and  the  Web  publishing.  At  the  time,  this  seemed  like  a 
natural  division  of  the  project.  The  two  vendors  estimated 
that  the  entire  project  could  be  completed  for  approximately 
$1.6  million.  This  was  a  little  under  our  anticipated  total  esti¬ 
mate,  but  we  were  aware  that  we  had  only  provided  the  ven¬ 
dors  with  an  overview  of  the  customizations  we  might  require. 


The  vendors’  promises  looked  too  good  to  be  true. 
As  it  turned  out,  they  were. 


NCCI  asked  the  content  management  vendor  to  give  us  a 
workflow  demonstration  using  one  of  our  manuals.  We  also 
asked  the  Web  publishing  vendor  to  show  us  how  the  manual 
would  be  published  on  our  website.  The  document  manage¬ 
ment  vendor  built  a  very  easy  workflow,  walked  our  internal 
users  through  it,  and  talked  about  how  easy  it  would  be  to 
check  the  manual  and  send  it  through  a  simple  workflow 
process  as  approvals  and  changes  were  made  to  the  document. 
The  Web  publishing  vendor  took  the  sample  manual  and 
demonstrated  how  to  publish  it  on  our  website.  The  two  ven¬ 
dors  were  confident  that  we  would  not  need  ongoing  IT  sup¬ 
port  for  the  internal  users.  The  system,  they  said,  “Was  easy 
enough  for  any  user  to  just  go  through  the  process  without 
much  guidance  or  support.” 

It  looked  too  good  to  be  true.  It  was. 

What  Due  Diligence  Revealed 

While  the  vendors  were  able  to  show  the  document  moving 
through  a  workflow,  they  were  not  able  to  go  through  all  of 
the  steps  needed  to  actually  publish  the  document.  Their  expla¬ 
nation  was  that  publishing  would  require  some  configuration 
and  customization,  and  take  more  time  than  we  had  allotted  for 
the  proof-of-concept  stage.  But  they  insisted  we  would  have 
both  systems  up  and  running  within  weeks. 

During  this  process,  representatives  from  both  vendors  were 
setting  up  alliances  with  folks  on  the  selection  team  and  lob¬ 
bying  for  their  respective  systems.  The  Web  publishing  vendor, 
in  particular,  became  very  friendly  with  some  of  our  users.  Its 
salespeople  had  these  users  convinced  that  its  software  was  the 


perfect  choice  for  our  project.  Needless  to  say,  this  made  price 
negotiations  with  the  vendor  difficult. 

At  this  point,  the  president  and  CFO  became  more  actively 
involved  and  decided  NCCI  should  do  one  more  check  with  a 
trusted  consultant  to  more  closely  examine  the  project’s  require¬ 
ments  and  costs.  I  called  upon  Avenue  A/Razorfish,  a  New 
York-based  consultancy  that’s  helped  us  build  Web-based  appli¬ 
cations.  After  studying  the  project  in  detail,  Avenue  A/Razorfish 
identified  all  the  customizations  we  would  have  to  do  with  the 
document  management  system — additional  work  that  the 
vendor  had  somehow  neglected  to  mention.  Because  of  these 
customizations,  the  consultants  showed  us  why  the  new  system 
would  cost  significantly  more  than  the  original  estimate. 

By  this  time,  we  had  become  aware  of  the  difficulty  of  try¬ 
ing  to  manage  two  different  vendors  on  what 
should  have  been  a  seamless  enterprise  proj¬ 
ect.  We  needed  one  team  that  would  not  be 
pointing  fingers  at  someone  else’s  flaws.  The 
executive  committee  decided  it  was  not  in  our 
best  interest  to  have  an  open-ended  contract 
with  two  different  vendors. 

Instead,  NCCI  decided  to  go  with  a  fixed-bid  contract  of 
$2.1  million  with  Avenue  A/Razorfish;  aside  from  being  one  of 
our  consulting  partners,  it  is  also  one  of  our  preferred  vendors. 
We  felt  that  the  fixed  bid  allowed  us  to  better  plan  our  budg¬ 
eting  and  helped  us  to  control  any  project  “creep.” 

The  project  is  now  moving  forward,  and  we  have  had  some 
good  interim  results.  In  addition,  the  cost  of  the  project  is  on 
track,  and  the  internal  team  is  now  up  to  speed.  We  feel  con¬ 
fident  that  we  will  be  able  to  move  forward  without  the 
consulting  group  when  we  add  new  manuals  or  workflows. 

In  the  end,  we  learned  several  valuable  lessons  that  may  be 
of  use  to  other  organizations  as  well: 

■  Limit  the  vendors’  access  to  your  organization  until  you 
have  finished  the  negotiations  and  finalized  your  contract. 

■  Always  get  a  second  opinion  on  technology  that  you  are 
not  familiar  with,  and  solicit  opinions  from  experts  other  than 
the  vendors  you  are  buying  the  technology  from. 

■  When  you  evaluate  a  proof-of-concept  demonstration, 
make  sure  you  ask  about  processes  that  might  require  cus¬ 
tomization.  Then,  ask  again  and  again  during  each  step. 

■  Make  sure  your  top  executives  are  involved  in  the  project 
right  from  the  get-go.  Otherwise,  you’ll  spend  a  lot  of  time 
reinventing  the  wheel.  HH 


Jeff  Fields  is  CIO  of  Customer  Operations  at  NCCI 
and  is  responsible  for  all  client  and  business  applica¬ 
tions  for  customer  operations,  e-commerce,  and 
NCCI's  website  and  regulatory  systems.  He  can  be 
reached  at  Jeff_Fields@ncci.com. 
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It  goes  against  human  nature  to  always  expect  the 
worst.  But  with  IT  projects,  pessimism — otherwise 
known  as  contingency  planning — is  the  only  way  to 
keep  small  technology  problems  from  becoming  full¬ 
blown  business  disasters. 

Too  bad  no  one  can  bring  themselves  to  do  enough  of  it. 
Christina  Hanger  had  little  reason  to  be  pessimistic  in  May 
2004,  when  she  was  moving  one  of  Hewlett-Packard’s 
biggest  North  American  divisions  onto  a  centralized  ERP 
system  from  SAP.  As  the  leader  of  an  IT  consolidation 
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In  retrospect,  Gilles  Bouchard,  HP 
CIO  and  executive  VP  of  global 
operations,  says  he  should  have  had 
additional  manufacturing  capacity 
ready  before  the  ERP  rollout. 
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project  rooted  in  HP’s  acquisition  of  Compaq  two  years  earlier, 
Hanger,  HP’s  senior  vice  president  of  Americas  operations 
and  IT,  had  an  unbroken  record  of  success  migrating  five  prod¬ 
uct  groups  within  the  two  former  companies  onto  one  of  two 
SAP  systems. 

Hanger  had  every  reason  to  believe  that  the  sixth  would  go  well 
too.  Even  so,  she  knew  to  be  prepared  for  problems.  At  approxi¬ 
mately  $7.5  billion  in  annual  revenue,  the  division  involved  with 
this  latest  project,  Industry  Standard  Servers  (ISS),  is  much  larger 
than  any  of  the  others  that  Hanger  had  migrated  to  SAP  to  that 


new  SAP  system.  The  contingency  plan  addressed  business 
impacts  too.  HP  banked  three  weeks’  worth  of  extra  servers  and 
took  over  an  empty  portion  of  an  HP  factory  in  Omaha  to  stand 
by  for  any  overflow  of  orders  that  needed  special  configurations 
(for  example,  an  unusual  component  or  software  combination) 
and  could  not  be  stockpiled  ahead  of  time. 

But  the  plan  wasn’t  pessimistic  enough. 

Starting  when  the  system  went  live  at  the  beginning  of  June 
and  continuing  throughout  the  rest  of  the  month,  as  many  as 
20  percent  of  customer  orders  for  servers  stopped  dead  in  their 


“We  planned  for  three  weeks 
of  extra  inventory,  ”  but  demand 

spiked  35  percent,  and  the 

factory  was 
overwhelmed. 


tracks  between  the  legacy  order-entry  system  and  the  SAP  sys¬ 
tem.  As  IT  problems  go,  this  wasn’t  too  big:  Some  data  modeling 
issues  between  the  legacy  system  and  the  SAP  system  prevented 
the  SAP  system  from  processing  some  orders  for  customized 
products.  These  programming  errors  were  fixed  within  four  or 
five  weeks.  But  Hanger  and  her  business  colleagues  from  the  ISS 
division  who  were  on  the  project  steering  committee  never  envi¬ 
sioned  the  degree  to  which  these  programming  glitches  would 
affect  the  business. 

Orders  began  to  backlog  quickly  and  HP  did  not  have  enough 
manual  workarounds  to  keep  servers  flowing  fast  enough  to  meet 
customer  demand.  Angry  customers  picked  up  the  phone  and 
called  HP — or  worse,  arch-competitors  Dell  and  IBM.  In  a  com¬ 
modity  market  such  as  servers,  customer  loyalty  is  built  upon  a 
company’s  ability  to  configure  products  to  order  and  get  them 
delivered  on  time.  HP  could  do  neither  for  much  of  the  summer. 
In  a  third-quarter  conference  call  on  Aug.  12,  HP  Chairman  and 
CEO  Carly  Fiorina  pegged  the  financial  impact  at  $  1 60  million: 
a  $120  million  order  backlog  that  resulted  in  $40  million  in  lost 
revenue.  That’s  more  than  the  cost  of  the  project  itself,  which 


-CHRISTINA  HANGER,  SENIOR  VP  OF 
AMERICAS  OPERATIONS  AND  I.T,  HP 


point.  So  Hanger  took  the  contingency  plan  that  her  team  had 
developed  for  the  other  five  migrations  and  adjusted  it  to  accom¬ 
modate  the  ISS  division’s  larger  sales  volume.  She  planned  for  three 
weeks  of  IT  snafus,  mostly  focused  on  what  might  happen  as  a 
result  of  tweaking  a  legacy  order-entry  system  to  work  with  the 
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AMR  Research  estimates  to  be  $30  million. 

The  headlines  all  claimed  an  IT  disaster,  but  in  fact,  HP’s  dis¬ 
aster  resulted  from  a  few  relatively  small  problems  in  IT  that 
snowballed  into  a  much  bigger  problem  for  the  business:  the 
inability  to  cope  with  the  order  backlog.This  was  a  disaster  that 
could  have  been  prevented — not  by  trying  to  eliminate  every  pos¬ 
sibility  for  error  in  a  major  IT  system  migration,  which  is  virtually 
impossible,  but  by  taking  a  much  broader  view  of  the  impact  that 
these  projects  can  have  on  a  company’s  supply  chain. 

CIOs  don’t  run  the  supply  chain  in  most  companies,  so  they 
have  trouble  envisioning  what  will  happen  to  sales  if  a  critical 
system  doesn’t  function  as  expected  for  a  few  days  or  weeks.  Busi¬ 
nesspeople,  meanwhile,  have  trouble  imagining  an  IT  program¬ 
ming  glitch  getting  past  the  walls  of  the  data  center  and  causing 
hundreds  of  millions  of  dollars’  worth  of  lost  sales.  The  chasm 
between  cause  and  effect  is  almost  too  vast  to  contemplate. 

But  if  CIOs  want  to  stop  being  held  liable  for  hundreds  of  mil¬ 
lions  of  dollars  in  losses  for  relatively  small  IT  problems,  they 


1  Three  Steps  to  a  Business 

Contingency  Plan 

1 

CREATE  A  CROSS-FUNCTIONAL  TEAM  to  engage  business- 
people  and  educate  them  about  the  supply  chain  risks  of  a 
major  system  rollout. 

2 

DEVELOP  A  TRANSITION  PLAN  to  the  new  system  that 
assumes  the  system  will  fail  during  final  rollout.  Create  a 
conservative  time  estimate  for  the  period  it  could  be  down. 

3 

DEVISE  MANUAL  PROCESSES  for  keeping  orders  and  deliv¬ 
eries  flowing  during  the  problem  period.  Have  extra  people 
and  factory  capacity  on  call  to  handle  the  extra  workload. 

-C.K. 

have  to  convince  business  leaders  of  the  vastly  increased  risks  that 
major  enterprise  software  projects  pose  to  businesses  with  high- 
volume  supply  chains.  They  must  use  that  awareness  to  build — 
with  full  support  and  cooperation  from  the  business — a  business 
contingency  plan  for  IT  projects  that  is  as  robust  as  the  project 
plans  they  create  for  the  new  software. 

“The  potential  benefits  to  the  supply  chain  are  much  bigger  than 
the  IT  costs  in  projects  like  this,”  says  Bill  S wanton,  vice  president 
of  research  for  AMR  Research.  “But  the  potential  risk  to  the  supply 
chain  is  also  much  bigger.”  If  business  contingency  planning  con¬ 
tinues  to  play  a  secondary  role  to  IT  project  management,  the  prob¬ 


lem  is  only  going  to  get  worse  as  computer  systems  become  more 
powerful,  integration  methods  improve,  and  companies  consoli¬ 
date  their  critical  business  processes  on  fewer  and  fewer  systems. 

In  retrospect,  HP  CIO  and  Executive  Vice  President  of  Global 
Operations  Gilles  Bouchard  does  not  see  the  data  modeling  prob¬ 
lems  between  the  legacy  and  SAP  systems  as  the  source  of  the 
$160  million  impact.  He  focuses  on  HP’s  inability  to  keep  pace 
with  orders  in  the  supply  chain  once  the  problems  were  discov¬ 
ered.  “It  was  mostly  capacity  issues,  material  issues  and  factory 
issues,  ”  he  says.  “We  had  a  series  of  small  problems,  none  of  which 
individually  would  have  been  too  much  to  handle.  But  together 
they  created  the  perfect  storm.  ” 

The  Limits  of  Project 
Management 

When  contingency  planning  prevents  a  disaster,  it’s  nearly  impos¬ 
sible  to  tell  whether  everything  that  was  done  was  necessary. 
Remember  Y2K?  Many  business  leaders  continue  to  suspect  that 
the  billions  spent  on  Y2K  was  a  waste  of  money  because  nothing 
happened.  CIOs  who  try  to  warn  their  CEOs  that  programming 
glitches  could  cost  millions  will  almost  invariably  be  met  with 
this  response:  Then  make  sure  the  glitches  don’t  happen! 

That’s  why  IT  project  management  has  become  high  art  while 
business  contingency  planning  remains  in  the  Dark  Ages.  But  the 
problems  that  can  affect  an  enterprise  software  project  increase  all 
the  time  as  the  projects  encompass  more  code  and  more  business 
processes.  And  there  are  nearly  infinite  combinations  of  small  prob¬ 
lems  that  together  can  have  devastating  effects.  In  the  end,  it  is  riskier 
for  companies  to  try  to  protect  themselves  from  glitches  in  enter¬ 
prise  software  projects  through  ever  better  IT  project  management 
techniques  than  it  is  to  plan  for  manual  workarounds  to  get  prod¬ 
ucts  to  customers  in  case  of  failure. 

Other  companies  besides  HP  have  faced  similar  business  dis¬ 
asters  from  relatively  small  IT  errors.  Nike,  for  example,  had  a 
problem  with  a  demand-planning  application  when  it  switched 
to  a  centralized  SAP  system  in  2001.  The  problem  was  tamed 
within  a  few  weeks.  But  because  the  company  did  not  have  an 
adequate  business  contingency  plan,  the  small  glitch  in  IT  cost 
Nike  $100  million  in  revenue.  (See  the  online  version  of  “Nike 
Rebounds”  at  www.cio.com/prinklinks.) 

In  both  Nike’s  and  HP’s  cases,  the  IT  problems  were  due  to  a 
combination  of  factors  that  would  have  been  difficult  to  elimi¬ 
nate  in  the  project-planning  process.  At  HP,  Hanger  says  her  team 
tested  the  connections  between  the  legacy  front-end  ordering 
system  and  the  SAP  system.  And  the  connections  worked  fine  for 
orders  that  did  not  involve  any  custom  configuration,  as  well  as 
for  some  custom  orders. 

But  Hanger’s  team  was  unable  to  adequately  test  orders  that 
could  be  configured  by  customers  because  the  product  market- 
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ing  team  had  not  fully  scoped  the  breadth  of  configurations  that 
customers  would  want.  When  the  system  went  live,  some  of  these 
custom  configurations  went  through  and  others  did  not.  Those 
that  didn’t  got  spat  out  into  a  dead  zone,  sitting  idle  until  they 
could  be  entered  manually.  “We  had  customers  wanting  a  little 
more  flexibility  in  how  they  purchased  and  configured  their 
servers,”  she  says.  “And  we  did  not  always  have  the  data  mod¬ 
eled  correctly  [for  that  to  happen].”  Could  Hanger  have  tested 
more  configurations  prior  to  going  live?  Yes.  Could  the  team 
have  envisioned  all  of  them?  Probably  not. 

But  that  wasn’t  the  only  problem.  Hanger’s  team  trained  the 
customer  service  representatives  who  would  be  using  the  new 
system  two  weeks  prior  to  going  live.  All  representatives  were 
required  to  pass  a  test  showing  that  they  could  enter  orders  with¬ 
out  making  errors.  But  when  the  system  went  live,  the  represen¬ 
tatives  had  trouble  remembering  all  the  training  and  were 
flustered,  compounding  the  number  of  dropped  orders.  Hanger’s 


the  slack  two-to-three  weeks  later,  but  by  then,  it  was  too  late. 

Once  the  orders  backed  up,  the  only  way  HP  could  respond 
was  by  speeding  up  deliveries.  Depending  on  the  size  of  the  prod¬ 
uct  and  distance  shipped,  the  extra  cost  ranged  between  35  per¬ 
cent  and  40  percent,  according  to  HP,  and  gained  the  company  a 
few  days  on  orders  that  had  already  been  delayed  by  weeks. 

The  Contingency  Plan 
that  Wasn’t 

By  traditional  IT  contingency  planning  standards,  PIP  had  already 
gone  beyond  the  call  of  duty.  Most  IT  contingency  planning 
focuses  on  preparing  extra  code  rather  than  extra  products.  Con¬ 
vention  says  that  to  ensure  a  problem-free  transition  to  a  new  sys¬ 
tem,  there  should  be  a  redundant  system  ready  to  handle  things 
if  the  rollout  goes  sour.  At  the  very  least,  there  should  be  a  “roll¬ 
back”  strategy  to  go  back  to  the  old  system  if  there  is  an  issue. 


team  offered  refresher  training  two  weeks  into  the  rollout,  but 
by  then,  too  many  orders  had  fallen  out  to  catch  up.  “We  might 
have  improved  things  if  we  had  started  giving  them  refresher 
training  a  week  into  the  rollout  instead  of  two  weeks,”  she  says. 

Things  only  got  worse  when  customer  demand  for  configure- 
to-order  systems  spiked  by  35  percent  in  June,  beyond  what  HP’s 
demand  forecast  models  predicted.  Instead,  Hanger’s  business 
contingency  plan  called  for  normal  demand  through  the  summer. 
“We  had  planned  for  three  weeks  of  extra  inventory  at  a  50/50 
split  between  standard  servers  and  configure-to-order  servers,” 
she  says.  The  factory  in  Omaha  that  had  set  aside  some  of  its  lines 
to  handle  the  three  weeks  of  orders  quickly  became  overwhelmed. 
HP  rushed  to  get  a  second  factory  in  Europe  online  to  help  take  up 


But  both  Bouchard  and  Hanger  maintain  that  neither  strat¬ 
egy  would  have  worked  in  HP’s  case,  considering  the  scale  of  its 
server  business.  “If  we  had  had  two  systems  running,  that  would 
have  meant  that  every  supplier  would  have  had  an  order  for  us  in 
the  old  system  and  the  new  system,”  says  Hanger.  “When  it  was 
received  into  the  manufacturing  line,  it  would  have  had  to  be 
received  twice  and  then  [be  reconciled]  twice.”  Says  Bouchard: 
“You  would  have  created  a  backlog  of  orders  because  of  the  cum¬ 
bersomeness  of  the  duplication  effort.” 

What  HP  should  have  done  was  to  create  a  plan  for  taking 
orders  and  shipping  products  that  assumed  the  IT  system  it  had 
planned  to  use  didn’t  exist.  “  Contingency  planning  is  not  about 
IT,”  says  AMR  Research’s  Swanton.  “It’s  [about]  having  peo- 
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KfJT|lI|  To  manage  this  much  money,  you  need  a  solution  that  offers  optimum  performance  and  real-time  flexibility.  Microsoft 
found  both  in  the  AMD  Opteron™  processor  running  Windows  Server™  2003,  Enterprise  Edition.  The  AMD  Opteron 
processor  with  its  Direct  Connect  Architecture  already  powers  the  world’s  highest  performing  2-way  and  4-way  x86 
UOSiUl  servers.  Now  Microsoft  Treasury  managers  are  using  it  to  slash  the  time  it  takes  to  run  their  complex  risk-analysis 
software.  Meanwhile,  they  have  the  flexibility  to  go  from  32-  to  64-bit  applications  without  disrupting  their  business.  At  AMD,  we 
believe  it’s  critical  that  technology  should  migrate  on  your  terms  to  help  you  realize  your  unique  vision.  It’s  one  of  the  reasons  why 
leading  OEMs  offer  enterprise-class  solutions  powered  by  AMD  Opteron  processors.  Would  you  like  to  learn  just  how  much  of  a 
difference  they  can  make  to  your  company?  Go  to  www.amd.com/enterprise 
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pie  who  are  watching  what’s  happening,  able  to  detect  if  there’s 
a  problem  and  working  out  some  simple  manual  way  around  it 
until  you’re  ready  to  work  with  the  system.  If  that  takes  a  team 
working  a  bunch  of  overtime,  fine.  It  will  be  a  lot  less  disruptive 
than  losing  sales.”  Taking  your  order-to-cash  process  back  to 
the  1950s  requires  a  mind-set  change  among  all  the  people 
involved  in  that  process,  from  customer  service  representatives 
to  warehouse  clerks,  because  it  will  feel  silly  and  unnecessary — 
sort  of  like  imagining  what  will  happen  if  the  sun  doesn’t  come 
up  tomorrow. 

Next  Time,  Imagine 
the  Worst 

Even  extending  HP’s  business  contingency  plan  to  bank  an  addi¬ 
tional  few  weeks’  worth  of  servers  might  have  seemed  risky 
because  those  extra  servers  might  not  have  sold.  In  most  compa¬ 
nies  facing  such  decisions,  CIOs  feel  more  comfortable  trying  to 
eliminate  project  risk  by  perfecting  their  project  management 
skills.  This  route  appeals  to  our  optimism  and  quest  for  compe¬ 
tence,  says  Robert  Charette,  president  of  Itabhi,  a  risk  manage¬ 
ment  consultancy.  Business  contingency  planning,  on  the  other 
hand,  is  gloomy  and  expensive. 

But  if  companies  are  ever  going  to  perfect  IT  project  manage¬ 
ment,  Charette  says,  it  will  require  a  revolution  to  overthrow  the 
principles  that  rule  most  IT  projects:  budget  and  schedule.  “Cost 


How  to  Avoid  a  $120  Million  Order  Backlog 

Hewlett-Packard’s  contingency  plan  accounted  for  the  business  impact 
of  software  snafus,  but  didn’t  contemplate  the  worst-case  scenario 

What  HP  Did 

What  HP  Should  Have  Done 

TRAINING 

Began  refresher  training  in  the 
second  week  after  rollout 

Begin  refresher  training  in  the  first 
week  of  the  rollout 

MANUFACTURING 

Stockpiled  three  weeks’  worth 
of  servers 

Stockpile  five  or  six  weeks’  worth 
of  servers 

Added  capacity  only  at  Omaha 
factory  prior  to  rollout 

Have  additional  capacity  ready 
elsewhere  before  the  rollout 

CUSTOMER  SERVICE 

Manually  entered  orders  into 
system  two  weeks  after  rollout, 
when  problems  with  data 
model  were  fixed 

Devise  manual  order-taking 
process  to  be  used  until  system 
problems  were  fixed 

Added  more  customer  service 
reps  when  orders  piled  up 

Train  additional  customer  service 
reps  prior  to  rollout  so  that  they 
would  be  available  to  tackle  backlog 

and  schedule  are  what  let  people  rationalize  away  crucial  pieces  of 
project  management  like  application  testing  and  training,”  he  says. 

Worse,  a  cost-and-schedule  approach  never  holds  up  during  a 
crisis,  says  Charette.  When  HP  saw  that  the  order  management 
system  wasn’t  working  properly,  it  pulled  out  all  the  stops  to  get  the 
code  working  properly — cost  and  schedule  be  damned.  Charette 
says  when  using  this  event-driven  approach,  “the  project  doesn’t 
move  forward  until  you’ve  gotten  each  step  right.”  Yet  event- 
driven  project  management  has  its  own  pitfalls.  It’s  not  infallible, 
and  if  not  carefully  managed,  projects  can  drag  on  forever. 

Here’s  what  Bouchard  would  have  done  differently.  He  would 
have  expanded  the  contingency  plan  to  cover  five  or  six  weeks. 
Instead  of  trying  to  prevent  IT  problems  that  were  too  small  and 
rolled  up  in  too  many  strange  combinations,  it  would  have  been 
easier  to  bring  the  backup  factory  in  Europe  online  earlier  and 
stockpile  more  generic  servers.  In  his  interview  with  CIO,  he  did 
not  address  whether  additional  manual  workarounds,  such  as  a 
manual  order-entry  process,  would  have  helped. 

If  that  sounds  like  passing  the  buck,  Bouchard  is  only  passing 
it  from  one  hand  to  the  other.  In  December  2003,  he  became 
CIO  and  executive  vice  president  of  global  operations — one  of 
those  rare  CIOs  who  also  runs  the  supply  chain.  Besides  empha¬ 
sizing  business  contingency  planning  more  strongly,  he  is  in  the 
process  of  reorganizing  the  operations  and  IT  groups  of  HP’s 
businesses.  Bouchard  replicated  his  dual  role  at  the  regional  level 
too.  Hanger  runs  both  IT  and  operations  for  the  Americas.  The 

more  consolidated  approach 
should  improve  communica¬ 
tion  between  IT  and  the  busi¬ 
ness,  he  believes,  and  make  it 
easier  to  identify  how  IT  proj¬ 
ects  affect  operations. 

One  message  that  needs  to  be 
communicated  more  strongly 
within  HP — and  within  every 
company  these  days,  Bouchard 
believes — is  the  message  that  is 
implicit  in  his  dual  role:  “There 
is  big  leverage  between  IT  and 
the  business  processes  when  you 
deal  with  a  large  supply  chain,” 
he  says.  “Just  looking  at  contin¬ 
gency  planning  from  an  IT  point 
of  view  would  be  a  big  mistake. 
It  has  to  be  looked  at  from  an 
integrated  view  of  IT  processes 
and  the  business.”  HPl 


Contact  Executive  Editor  Christo¬ 
pher  Koch  at  ckoch@cio.com. 
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Protect  your  revenues  as  well  as  your  software. 


Are  you  giving  away  one  copy  of  your  software  for  every  two  that  you  sell?  It's  quite  likely. 
Because  a  recent  study  shows  that  only  two  of  every  three  software  applications  in  use  today 
were  purchased  legally.*  So  if  you  can’t  afford  to  give  up  a  third  of  your  revenue,  get  SafeNet 
Sentinel.  We’ve  been  protecting  software  developers  against  piracy  and  hackers  for  more 
than  15  years.  We’ll  help  manage  your  rights  -  so  you  get  what’s  rightfully  yours. 


Call  1-800-533-3958  to  be  SafeNet  sure. 

www.safenet-inc. com/sentinel-500 

Copyright  2004,  SafeNet,  Inc.  All  rights  reserved.  SafeNet  and  Safe  Net  logo  are  registered  trademarks  of  SafeNet,  Inc. 
*First  Annual  BSA  and  /DC  Global  Software  Piracy  Study,  July  ‘04.  (NASDAQ:  SFNT) 


The  Foundation  of  Information  Security 


J.  Clark  Kelso  (center),  California’s  CIO  and  a  veteran  IT 
troubleshooter  for  the  state,  is  adept  at  maneuvering  through 
the  corridors  of  political  power.  Here,  he  chats  with  Deputy 
State  CIOs  Rob  Quigley  (left)  and  Daniel  Gullahorn  at  the 
California  State  Capitol  in  Sacramento. 


is  a  CIO  with  no 
central  IT  department  and  little 
executive  authority.  Vet  with  his  skill  for 
reading  situations  and  people,  he  has 
calmed  a  crisis  and  brought  the  state’s 
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Reader  ROI 

►  How  California’s  CIO 
survived  the  recall  election 
and  Schwarzenegger’s  new 
regime 

What  he  has  done  to 
begin  turning  the  state’s 
IT  around 

Why  his  success  (thus  far) 
is  an  instructive  tale  for 
CIOs  everywhere 


Profile 


J.  Clark  Kelso 


While  Kelso  (front)  has  made  a  name  for  himself  as  a  good  listener  and  mediator,  he  also 
knows  how  to  take  charge.  Here,  he  discusses  strategy  with  Roy  McBrayer,  interim  adviser  to 
the  state  CIO,  at  a  recent  meeting  in  the  Governor's  Office  of  Planning  and  Research. 


"‘The  IT  community  was  feeling  badly 
burned  by  the  Oracle  scandal,”  says  Carlos 
Ramos,  who  attended  the  gathering  as  then- 
head  of  the  Stephen  R  Teale  Data  Center. 
“Clark  brought  calm  and  reconciliation.  He 
encouraged  us  to  move  ahead.” 

With  that  first  meeting,  Kelso,  soon  to  be 
appointed  CIO  of  the  country’s  most  popu¬ 
lous  state,  set  the  stage  for  rebuilding  trust  and 
confidence  in  the  fractured  world  of  Califor¬ 
nia  IT.  The  45-year-old  legal  scholar  is  a  per¬ 
suasive,  often  forceful,  speaker;  yet  he  listens 
carefully  at  meetings  and  comes  across  as  open 
and  unassuming.  He’s  a  CIO  with  no  central 
IT  department  and  little  executive  authority. 
Yet  with  his  skill  for  reading  situations  and 
people,  he  has  brought  the  state’s  far-flung  and 
often  feuding  IT  leaders  together,  making 
progress  on  consolidation  projects  that  have 
languished  for  years.  He  has  also  forged  ties 
with  a  skeptical  state  legislature,  which  has 
been  burned  by  a  string  of  failed  IT  projects. 
“He  doesn’t  come  in  like:  I’m  King  Kong,  and 
I  have  all  the  answers,”  says  Ramos,  who  is 
now  assistant  secretary  of  the  state’s  Health 
and  Human  Services  Agency. 

Kelso  also  appears  to  possess  an  uncanny 
knack  for  survival.  Though  he  was  appointed 
in  2002  by  Democratic  Gov.  Davis,  he  is  one 
of  few  department-level  heads  to  gain  the  con¬ 
fidence  of  current  Republican  Gov.  Arnold 
Schwarzenegger,  who  swooped  into  power 
after  the  2003  recall  election.  (Kelso  is  a  regis¬ 
tered  Republican,  but  says  he  considers  him¬ 
self  nonpartisan  and  politically  independent. ) 

Armed  with  such  powerful  allies,  Kelso  has 
been  able  to  move  forward  with  plans  to 
restart  e-government  initiatives  and  wring  effi¬ 
ciencies  from  IT  procurement  practices  that, 
until  now,  have  been  heavily  redundant.  He 
has  also,  with  help  from  a  council  of  state  IT 
leaders  he  assembled,  drafted  a  five-year 
strategic  plan  for  California  IT  that  will  guide 
the  replacement  of  the  state’s  aging  legacy  sys¬ 
tems  for  finance  and  other  back-office  func¬ 
tions  with  integrated  ERP-type  systems.  In  a 
world  where  CIOs  rarely  stay  in  the  same 
position  for  more  than  two  years,  Kelso’s  suc¬ 
cess  in  calming  a  political  crisis,  coaxing  rivals 
to  work  together  and  hanging  on  in  a  fiercely 


divided  political  climate  is  an  instructive  tale. 

“Clark  is  at  heart  a  public  servant  and  I 
think  everyone  knows  that,”  says  state  Sen. 
Debra  Bowen  (D-Redondo  Beach).  Bowen 
says  that  Kelso  has  passed  a  simple  test  for  suc¬ 
cess  in  California  politics:  Since  he  has  taken 
over,  the  scathing  headlines  about  state  IT 
have  all  but  disappeared  from  local  newspa¬ 
pers.  She’s  not  surprised  that  Schwarzenegger 
has  kept  him  on.  “With  Clark,  you  don’t  have 
to  worry  about  any  hidden  agendas,”  she  says. 

A  Lawyerly  Approach 

Ensconced  in  his  Sacramento  office,  crowded 
with  stacks  of  books,  documents  and  framed 
photos  of  his  family,  Kelso  appears  an  unlikely 
savior  for  California  IT.  He  is  a  lawyer  by 
training  and  still  teaches  government  law  two 
times  a  week  at  McGeorge  School  of  Law, 
where  he  heads  the  Capital  Center  for  Gov¬ 
ernment  Law  and  Policy.  Although  he  has 


long  taken  an  interest  in  computers,  he  has  no 
formal  IT  training.  Kelso  is  stocky  with  a 
closely  cropped  beard,  and  though  he  has  a 
generally  serious  demeanor,  he  is  given  to 
short  and  sudden  bursts  of  laughter.  In 
between  conference  calls  with  pushy  technol¬ 
ogy  vendors  and  distant  agency  CIOs,  he 
keeps  his  cool  by  playing  Chopin  piano  con¬ 
certos  and  Mozart  arias  on  a  stereo  in  his 
office,  which  is  housed  several  miles  from  the 
state  capital  in  the  Sacramento  law  school. 
(He  prefers  to  work  out  of  his  law  office 
because  he  has  a  personal  assistant  there.)  His 
varied  attire  attests  to  his  chameleonlike  abil¬ 
ity  to  fit  into  different  situations:  When  con¬ 
ducting  meetings  by  phone  and  hanging 
around  his  law  school  office,  he  wears  casual, 
short-sleeved  shirts.  On  a  day  when  he  is  head¬ 
ing  to  the  governor’s  office,  however,  he  is  dap¬ 
per  in  a  dark  gray  pinstripe  suit,  suspenders 
and  gold  JCK  cufflinks. 

On  a  warm,  breezy  day  in  September, 
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Now  you  can't  say  you  haven't  heard  of  us 


Now 


that  you've  heard  of  us,  here  are  six  good  reasons  why  you  should  care: 


•  Equant  is  the  proven  industry  leader  in  global  business  communication  solutions  for  multinational 
companies. 

•  Supporting  you  with  people  and  expertise  in  165  countries,  wherever  you  do  business. 

•  A  single,  seamless  network  in  220  countries  and  territories. 

•  Saving  you  time  and  money,  improving  end-user  and  customer  satisfaction,  delivering  solutions 
tailored  to  your  business  processes. 

•  Offering  you  a  complete  range  of  global  communication  solutions,  addressing  your  urgent  issues: 
Compliance,  Security,  Applications  Performance,  Vendor  Management,  IP  Telephony. 

•  Enabling  your  global  communications  to  run  smoothly  so  you  can  focus  on  your  company's  core 
business. 


To  discover  how  Equant's  solutions  can  enhance  your  business  visit  our  website  now: 

www.equant.com/usa. 


When  you  want  to  transform 

your  global  communications,  we'll  be  with  you 


<Oequant 


Profile 


J.  Clark  Kelso 


Kelso,  who  is  married  and  has  two  daughters, 
dashes  from  his  office  at  4:30  to  pick  up  his 
4-year-old  at  day  care.  He  is  devoted  to  his 
family  and  is  a  pushover  when  it  comes  to  his 
daughters.  In  his  professional  life,  however, 
Kelso  has  made  a  name  for  himself  as  a  tough 
crisis  manager.  In  2000,  state  Attorney  Gen¬ 
eral  Bill  Lockyer  and  Davis  tapped  Kelso,  then 
a  law  professor  active  in  court-related  IT  ini¬ 
tiatives,  to  oversee  the  state’s  Department  of 
Insurance  in  the  wake  of  a  political  scandal 
involving  the  resignation  of  then-insurance 
Commissioner  Chuck  Quackenbush.  Kelso 
arrived  after  criminal  and  legislative  investi¬ 
gations  alleged  Quackenbush  and  his  depart¬ 
ment  had  funneled  insurance  company 
settlement  money  into  foundations  that  ben¬ 
efited  Quackenbush  politically  and  others 
financially.  On  his  second  day  at  the  agency, 
Kelso  fired  seven  top  deputies. 

Davis  then  picked  Kelso  to  chair  the  Cali¬ 
fornia  Earthquake  Authority,  which  was  being 
investigated  by  the  Bureau  of  State  Audits  in 
the  aftermath  of  the  insurance  scandal. 

“If  you  do  something  once  around  here, 
they  put  a  label  on  you,”  says  Kelso.  “A  lot 
of  people  see  me  as  a  government  trou¬ 
bleshooter.  It’s  a  good  position  to  be  in  around 
here  because  there  is  almost  always  trouble,” 
Kelso  says. 

A  California-Sized 
Disaster 

The  state’s  IT  department  is  no  exception. 
The  state,  with  a  population  of  nearly  36  mil¬ 
lion,  employs  more  than  270,000  people  and 
is  run  by  11  agencies,  79  departments,  and 
more  than  300  boards  and  commissions — all 
working  independently,  many  of  them  with 
separate  CIOs.  Recurring  budget  deficits 
have  long  hobbled  the  state’s  ability  to 


upgrade  its  IT  infrastructure,  and  easy  solu¬ 
tions  don’t  appear  close  at  hand.  This  past 
July,  the  state  balanced  a  $105.4  billion 
budget  through  spending  cuts,  revenue  shuf¬ 
fles  and  debt  refinancing.  While  funding  is 
assured  for  major  IT  projects — such  as  the 
$100  million,  seven-year  21st  Century  Pro¬ 
ject  that  will  replace  that  state’s  homegrown 
system  for  payroll  with  ERP — discretionary 
IT  spending  is  down.  In  January,  Schwarzeneg¬ 
ger  will  present  a  new  budget  plan  that  will 
close  a  projected  $5.1  billion  deficit.  More 
spending  cuts  are  expected. 

Even  without  the  constant  budget  chal¬ 
lenges,  scandal  and  mismanagement  have 
long  plagued  California  IT.  In  the  1990s,  the 
state  suffered  a  string  of  high-profile  IT  deba¬ 
cles,  starting  with  a  botched  state  lottery  tech¬ 
nology  contract  in  1992  that  cost  the  state 
$52  million.  In  1994,  the  state  paid  $5 1  mil¬ 
lion  for  a  Department  of  Motor  Vehicle  data¬ 
base  that  never  worked.  In  1997,  the  state 
stumbled  yet  again,  abandoning  a  $1 1 1  mil¬ 
lion  system  to  track  child  support  after  county 
officials  called  it  unusable.  Most  of  these  proj  - 
ects  were  eventually  finished  successfully.  But 
those  involved  with  the  ill-fated  ventures 
blame  (in  part)  the  lack  of  central  IT  oversight 
to  approve  IT  proposals. 

“California  has  had  a  lack  of  vision  and 
leadership  when  it  comes  to  IT,”  says  Bowen. 
California  lags  behind  other  states,  such  as 
Utah,  where  former  Gov.  Mike  Leavitt  gave 
full  support  to  e-government  and  other  tech¬ 
nology  initiatives.  “We  haven’t  had  a  struc¬ 
ture  yet  with  a  strong  IT  leader  backed  by  the 
governor’s  office  to  deal  with  turf  wars 
between  agencies,”  Bowen  adds. 

Past  attempts  to  remedy  the  perceived  lack 
of  IT  leadership  have  failed.  The  Legislature, 
acknowledging  the  problem,  created  DOIT  in 


Birthplace:  Indianapolis,  Ind. 

Education:  BA  in  philosophy 
and  a  minor  in  mathematics,  Uni¬ 
versity  of  Illinois,  1980;  JD  from 
Columbia  University,  1983. 

First  Job:  System  programmer  at 
University  of  Illinois. 

First  government  job:  Consultant 
for  California  on  the  independence 
of  the  courts  and  the  appellate  jus¬ 
tice  system. 

Avocations:  Singing  comic  opera, 
writing  books  and  professional 
articles  about  law  and  ethics. 

Favorite  Hobby:  Cultivating  roses 
and  tulips. 


1995.  John  Thomas  Flynn,  the  state’s  first 
CIO,  appointed  by  Gov.  Pete  Wilson,  says  that 
the  state’s  data  centers  “ruled  the  roost”  with 
huge  budgets.  Getting  agency  leaders  to  move 
ahead  with  large-scale  projects  was  a  night¬ 
mare.  “I  never  ceased  to  be  amazed  by  the  lack 
of  action  on  issues,”  Flynn  says. 

From  one  administration  to  the  next, 
DOIT — with  69  employees  and  a  budget  of 
$11  million  in  2002 — never  managed  to 
reverse  negative  perceptions  about  Califor¬ 
nia  IT.  The  Oracle  scandal  was  the  last  straw. 
In  May  2001,  DOIT  and  the  state  Depart¬ 
ment  of  General  Services  signed  an  exclusive 
$95  million  deal  to  buy  Enterprise  Edition  8i 
database  software  from  Oracle.  A  state  audit 
subsequently  reported  that  the  no-bid  con¬ 
tract  would  have  saddled  the  state  with  mil¬ 
lions  of  dollars  of  unneeded  software.  Davis 
suspended  Cortez  in  May  2002,  the  contract 
was  canceled,  and  Kelso  was  appointed 
interim  head  of  DOIT.  Soon  after,  Cortez  was 
fired  and  disappeared  from  Sacramento  pol¬ 
itics.  (Attempts  to  reach  Cortez  were  not  suc¬ 
cessful.)  Legislators,  furious  with  state 


“A  lot  of  people  see  me  as  a  government 
troubleshooter.  It’s  a  good  position  to  be 
in  around  here  because  there  is  almost 
always  trouble.” 

-J.  Clark  Kelso,  CIO  of  the  state  of  California 


64  CIO  DECEMBER  1,  2004  •  www.cio.com 


Profile 


J.  Clark  Kelso 


officials,  pulled  the  plug  on  DOIT  on  July  1, 
2002.  The  state’s  grand  experiment  in  cen¬ 
tralizing  IT  was  no  more. 


Sacramento  Insider 

If  Sacramento  lawmakers  and  government 
officials  still  hate  IT,  it  isn’t  noticeable  on  this 
morning  in  September.  Kelso  steers  his  dark 
green,  1996  Isuzu  Trooper  down  the  state 
capital’s  wide  and  straight  boulevards,  from 
his  office  at  the  law  school  toward  the  hub  of 
the  state’s  political  activity.  First  stop:  the  Zig- 
gurat  Building,  a  pyramid-shaped  structure 
that  houses  the  Department  of  General  Ser¬ 
vices.  He  straightens  his  collar  before  entering 
a  meeting  on  “strategic  sourcing,”  a  govern¬ 
mentwide  plan  to  slash  costs  by  buying  IT 
equipment  and  other  state  supplies  on  a  mas¬ 
sive  scale.  This  fall,  he  says,  the  state  will  start 
the  first  enterprisewide  IT  procurement  since 
the  Oracle  scandal,  an  effort  he  says  will  save 
the  state  tens  of  millions  of  dollars  in  hard¬ 
ware  purchases  alone. 

“It  promises  to  be  a  fun  fall  around  here,” 
he  says  with  a  laugh,  making  light  of  the  com¬ 
plaints  he  expects  from  state  employees  about 


Population:  35.5  million 

State  employees:  270,000 

State  departments:  11  agen¬ 
cies,  79  departments  and 
more  than  300  boards,  and 
commissions 

State  budget:  $105.4  billion 

Output  of  goods  and  services: 

$1.4  trillion  (fifth  largest  in 
the  world  after  the  United 
Kingdom) 

IT  annual  spending:  roughly 
$2  billion 

Square  miles:  155,959 

Sources:  2003  Statistical  Abstract  of  the 
United  States,  California  Chamber  of 
Commerce,  and  J.  Clark  Kelso 


having  to  change  their  PCs  when  the  state 
replaces  them  all  with  one  model  as  a  part  of 
the  strategic  sourcing  plan. 

As  he  strolls  the  halls  of  the  Ziggurat  Build¬ 
ing,  receptionists,  lobbyists  and  fellow  gov¬ 
ernment  employees  greet  him  warmly, 
slapping  him  on  the  back  and  yelling  out,  “Hi 
Clark.  How’s  it  going?  ”  Later  that  day,  Kelso 
saunters  into  a  warrenlike  complex  that  leads 
to  the  governor’s  office,  greeting  staffers  and 


to  bring  down  the  general  stress  level. 
“[DOIT  employees]  were  scared  to  death  of 
losing  their  jobs  so  I  had  to  do  this  all  while 
maintaining  a  sense  of  calm,”  he  says.  Soon, 
the  crisis  subsided,  and  Kelso  set  out  to  cre¬ 
ate  a  plan  for  California  IT  with  the  collabo¬ 
ration  of  his  many  agency  CIOs. 

Many  of  Kelso’s  ideas  are  not  new.  One 
example:  the  recent  executive  order  by 
Schwarzenegger  (which  was  drafted  prima¬ 


“I  came  in  to  clean  up  a  mess  and  set  a 
course  for  the  future.  I  won’t  be  satisfied 
until  I  can  convince  the  Legislature  to 
create  an  office  of  the  CIO.” 

-J.  Clark  Kelso,  CIO  of  the  state  of  California 


sharing  inside  jokes.  Framed  glossy  photos  of 
Schwarzenegger  and  his  family  line  the  walls. 
The  atmosphere  is  relatively  calm,  Kelso 
notes,  because  Arnold  is  out  to  lunch  at  his 
favorite  local  watering  hole,  the  Esquire  Grill. 
When  the  governor  is  in,  Kelso  says,  the  place 
buzzes  with  activity,  and  staff  are  more  on 
edge.  Kelso  carries  a  stack  of  recommenda¬ 
tions  related  to  pending  state  legislation  on 
offshore  contracting  and  delivers  it  to 
Richard  Costigan,  Schwarzenegger’s  legisla¬ 
tive  affairs  secretary.  Kelso  is  worried  that  the 
bills  could  hurt  U.S.  competitiveness  in  the 
global  marketplace.  (Schwarzenegger  later 
vetoes  the  Democratic-sponsored  bill,  which 
would  have  prohibited  the  state  from  con¬ 
tracting  with  a  company  unless  all  of  the 
work  is  done  within  the  United  States.) 

Calm  in  a  Crisis 

When  faced  with  an  agency  in  turmoil,  Kelso 
is  quick  to  act.  At  DOIT,  he  began  by  being 
completely  forthright  with  the  press  and 
state  legislators.  He  consistently  returned 
reporters’  phone  calls,  for  example.  He  also 
began  working  with  legislative  liaisons  to  set 
up  meetings  with  key  members  of  the  state 
Senate  and  Assembly.  “You  can’t  appear  to 
be  hiding  anything,”  he  says.  He  also  worked 


rily  by  Kelso)  to  consolidate  the  state’s  two 
largest  data  centers.  The  two  data  centers,  one 
in  Sacramento  and  another  in  Rancho  Cor¬ 
dova,  run  most  of  the  state’s  applications  and 
have  combined  expenses  of  $300  million  a 
year.  Yet  they  have  long  been  run  as  separate 
fiefdoms.  The  idea  for  consolidation,  which 
could  save  the  state  $5  million  to  $7  million 
during  the  first  year,  has  been  floating  around 
for  years.  But  Kelso  “has  capitalized  on  the 
fact  that  IT  leaders  now  share  a  common 
vision,”  Ramos  says. 

Case  in  point:  At  an  afternoon  meeting  that 
same  day  in  September,  the  directors  of  the 
two  data  centers  being  consolidated  are  talk¬ 
ing  about  the  plan  with  Kelso  and  Ramos. 
Craig  Grivette,  the  director  of  the  Teale  Data 
Center,  outlines  a  plan  to  create  an  office  that 
will  manage  the  consolidation.  “We  need  peo¬ 
ple  from  both  data  centers  to  get  involved,” 
Grivette  says.  Bob  Austin,  his  counterpart  at 
the  state’s  other  main  data  center,  nods  in 
agreement.  Kelso  sits  back  and  listens  for 
much  of  the  meeting,  smiling  and  nodding  as 
the  two  data  center  leaders  amiably  share 
strategies  and  plans. 

After  the  meeting,  Kelso  runs  into  an  old 
acquaintance,  a  health-care  lobbyist  he  has 
worked  with  in  the  past.  “So  which  agency 
are  you  saving  these  days?”  she  asks. 
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Appointed  by  Democratic  Gov.  Davis,  Kelso  (right)  is  one  of  few  department-level  heads  to 
gain  the  confidence  of  current  Republican  Gov.  Arnold  Schwarzenegger. 


Hanging  Out  with 
Arnold 

Kelso  says  he  didn’t  know  what  to  expect 
when  Schwarzenegger  arrived  in  Sacramento. 
“It’s  a  curiosity  that  I  stayed  on,”  he  says.  “At 
this  point,  most  of  the  department  heads 
and  other  people  I  deal  with  are  Arnold’s 
appointees.”  Kelso  credits  his  perceived  abil¬ 
ity  “to  smother  a  crisis”  for  his  survival  so  far. 
Clearly,  Kelso  is  an  expert  political  operator 
who  knows  Sacramento’s  political  elite,  and 
who  knows  how  to  talk  to  lawmakers  while 
at  the  same  time  motivating  the  state’s  IT  lead¬ 
ers.  “Clark  is  always  interested  in  what  we’re 
doing,”  says  Jeff  Baldo,  CIO  for  the  state’s 
Youth  and  Adult  Correctional  Agency.  “He 
seems  to  get  along  with  everyone.  ” 

Schwarzenegger  himself  indicated  his 
confidence  in  Kelso  over  a  grilled  salmon 
lunch  at  the  Esquire  Grill  shortly  after  he 
took  over  as  governor:  He  asked  the  CIO  to 
draft  a  strategic  IT  plan  for  the  state  by  Nov  1 . 
“When  the  governor  asks  for  something,  you 
know  he  means  business,”  Kelso  says.  Put 
together  with  help  from  his  IT  council  of 
department  CIOs,  Kelso’s  plan  includes  cre¬ 
ating  a  new  state  e-government  portal  (since 
the  original  was  essentially  frozen  after  DOIT 
closed  shop)  and  updating  back-office  systems 
such  as  finance  and  HR  to  ERP  systems. 

Already,  he  says,  the  state  has  been  making 
slow,  steady  progress  on  isolated  IT  projects 
such  as  data  center  consolidation.  The  state 
has  also  started  work  to  consolidate  its  many 
e-mail  systems  (starting  with  25,000  seats) 
and  has  begun  procurement  for  a  project  to 
overhaul  the  state’s  payroll  system.  The  cur¬ 
rent  homegrown  system,  which  is  almost  30 
years  old,  poses  a  real  risk  to  state  operations 
because  the  only  person  who  really  knows 
how  to  fix  it  retired  last  year. 

A  recent  report  by  the  California  Perfor¬ 
mance  Review  (CPR),  a  commission  estab¬ 
lished  by  Schwarzenegger,  called  internal  IT 
systems  antiquated  and  fragmented,  and  rec¬ 
ommends  creating  a  centralized  department 
of  technology  services  and  a  CIO  position 
with  authority  to  direct  the  state’s  IT  invest¬ 
ments.  That  would  boost  the  position  to  Cab¬ 
inet  level  and  greatly  increase  the  CIO’s  power. 


Kelso  agrees  with  most  of  the  report’s  findings; 
although  he  wasn’t  on  the  commission,  he  was 
a  director  and  played  a  significant  behind-the- 
scenes  role  in  its  decision  making.  He  recom¬ 
mended  some  of  the  team  members  appointed 
to  the  commission. 

The  Once  and 
Future  CIO 

Despite  these  successes,  Kelso  has  his  work 
cut  out  for  him.  “People  would  like  to  see  the 
state  do  a  better  job  with  IT,”  Bowen  says.  For 
instance,  the  state  needs  to  look  at  “weird 
policies”  that  discourage  the  use  of  technol¬ 
ogy,  she  adds.  Until  recently,  if  you  renewed 
your  California  driver’s  license  online,  you 
paid  $4  more  than  if  you  did  it  on  paper,  even 
though  it  costs  the  state  less  money.  “It’s  just 
goofy,”  she  says.  Ultimately,  Bowen  says,  to 
address  these  sorts  of  issues,  the  state  CIO 
needs  full  backing  from  the  governor’s  office. 

Kelso  couldn’t  agree  more.  “I  came  in  to 
clean  up  a  mess  and  set  a  course  for  the 
future,”  Kelso  says.  “I  won’t  be  satisfied  until 
I  can  convince  the  Legislature  to  create  an 
office  of  the  CIO.”  If  that  happens,  will  he 


abandon  his  role  of  crisis  manager  to  take  on 
the  new,  more  powerful  CIO  role? 

Some  say  Kelso  is  the  best  person  for  the  job. 
“It  would  be  hard  to  bring  someone  in  from 
the  outside  at  this  point,”  says  Dave  Ross,  man¬ 
aging  partner  for  Accenture’s  state  and  local 
government  practice,  who  is  based  in  Sacra¬ 
mento.  “Clark  knows  how  it  all  works.”  But 
another  state  government  observer,  who  asked 
not  to  be  named,  counters  that  Kelso,  with  his 
diverse  interests  and  teaching  commitments, 
doesn’t  have  sufficient  time  for  the  job. 

It’s  not  clear  that  the  man  himself  would  be 
interested  in  this  new  role.  “I’m  not  focused 
on  that,”  Kelso  says,  leaning  back  in  his  chair. 
“Once  I  finish  the  strategic  plan  and  a  new 
CIO  office  is  created,  it  might  be  time  for 
someone  else  to  take  over.”  At  this  juncture, 
Kelso  is  focusing  on  concrete  projects  at  hand: 
data  center  consolidation  and  a  strategic  plan 
for  IT.  “I’ve  been  a  CIO  without  a  real  office 
and  without  a  department  for  the  past  two 
years,”  he  says.  “It’s  hard  to  imagine  doing  it 
any  other  way.”  H0 


Senior  Writer  Susannah  Patton  is  based  in  San  Fran¬ 
cisco  and  can  be  reached  at  spatton@cio.com. 
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I  I  Because  you  still  need 
to  demonstrate  the 
value  of  IT. 

We  offer  a  wealth  of 
different  perspectives, 
to  give  you  more  ways 
of  looking  at  value: 

The  Value  Proposition 
Tom  Davenport, 
Conference  Moderator, 
Professor  &  Director  of 
Research,  Executive  Education, 
Babson  College 

Why  is  it  still  so  hard  to  define  and 
demonstrate  the  strategic  value  of  IT 
to  the  enterprise?  Davenport,  noted 
consultant,  author  and  educator, 
kicks  off  the  conference  with  his 
insights  on  IT  value:  everyone  from 
the  business  unit  heads  to  the  chair¬ 
man  of  the  board  is  demandingfull 
valueforthe  organization's  IT  invest¬ 
ments.  How  do  you  agree  on  just 
what  the  value  will  be?  How  will  it  be 
measured?  How  do  you  best  com¬ 
municate  it  to  your  constituents? 


Chair  of  the  Executive  CIO 
Program,  Harvard  Business  School 

Austin  leads  us  through  a  case  cho¬ 
sen  for  its  emphasis  on  the  strategic 
value  of  IT  to  the  subject  enterprise. 
We  then  divide  into  small  working 
groups,  and  are  tasked  with  coming 
up  with  a  set  of  recommended 
actions  and  solutions  for  the  subject 
enterprise's  CIO.  Select  working 
groups  will  report  theirfindings  back 
to  the  assembly  at  large. 


View  From  the  Top: 

A  Perspective  on  IT 
Enterprise  Value 
William  J.  Shaw, 
President  &  COO,  Marriott 
International,  Inc. 

How  do  CEOs  and  other  senior 
management  think  about  the  role  of 
IT,  and  about  the  ability  of  the  CIO  to 
articulate  and  deliver  true  value  to 
the  enterprise?  What  criteria  do  they 
use  for  measuring  the  CIO’s  success 
orfailure?  Has  the  view  changed 
over  time— and  for  better  or  worse? 
What  were  the  major  influences  in 
changing  those  views?  We  listen 
as  a  top  executive  from  one  of  the 
Enterprise  Value  Award-winning 
companies  gives  us  his  “view  from 
the  top.” 


Conversations  with  This 
Year's  CIO  Enterprise 
Value  Award  Winners 

They're  first  scrutinized  by  CIO 
editors,  Review  Board  members,  and 
our  judging  panel  of  top-notch  CIOs. 
Meet  with  the  stars  of  this  year's  win¬ 
ning  organizations,  who  understand 
the  uphill  battle  IT  continues  to  face. 
It’s  your  chance  to  discuss  their  par¬ 
ticular  case  in  more  detail  and  take 
away  lessons  you  can  apply  to  your 
own  organization  back  home. 

*,»The  New  Imperative  for  Growth 
Many  businesses  have  maxed 
out  on  improving  productivity, 
increasing  efficiency  and 
decreasing  costs  as  a  way  to  affect 
the  bottom  line— and  it's  still  not 
enough.  The  new  imperative  is  for 
growth,  whether  organically  or  by 


acquisition.  The  role  of  the  CIO  in 
a  company's  growth  depends  on 
the  type  of  move  the  company  is 
making.  Are  you  shoring  up  exist¬ 
ing  business  by  strengthening  cus¬ 
tomer  ties?  Entering  new  markets? 
Acquiring  and  integrating  new 
businesses?  In  any  and  all  of  these 
cases,  IT  is  key.  Our  panel  of  sen¬ 
ior  IT  and  business  executives  con¬ 
siders  how  to  fuel  innovation  and 
enable  growth  in  a  business  envi¬ 
ronment  where  resources  are  still 
constrained. 

..^Extending  IT's  Reach 
Outside  the  Organization 
Several  of  this  year's  CIO 
Enterprise  Value  Award  Winners 
embody  a  growing  trend:  extend¬ 
ing  IT's  reach  further  outside  the 
enterprise,  into  deeper  relation¬ 
ships  with  supply  chains  and  cus¬ 
tomers  of  all  types.  Our  panelists 
share  what  they're  doing,  how  they 
make  it  work  (including  how  they 
safeguard  critical  company  data), 
and  what  the  payoff  is  in  real 
terms  (hint:  it's  impressive!). 

•  ■  ■  Mission-based  Value 

Our  definition  of  Enterprise  Value 
has  expanded  over  the  years,  but 
whatever  your  value  proposition, 
you  need  to  be  able  to  identify  it, 
build  it  and  measure  it.  While 
non-profit  organizations  certainly 
think  about  and  manage  their 
costs  as  tightly  as  any  business, 
they  think  about  value  not  in  terms 
of  profit  but  rather  in  terms  of  their 
enterprise's  core  mission.  Even  if 
your  organization's  value  doesn't 
come  from  drastically  altering  the 
quality  of  care  or  saving  lives,  you 
can  learn  how  to  think  about 
different  types  of  value  in  a  whole 
new  way. 


Speakers: 

Tony  Affuso,  Chairman, 

CEO  &  President,  UGS 

Chris  Baumann,  Manager, 
Transportation  &  Specialty 
Business,  Global  Systems  & 
Services,  ConocoPhillips 

Douglas  Caddell,  CIO, 

Foley  &  Lardner 

Kyle  Heath,  CMO,  Foley  &  Lardner 

John  Heller,  Vice  President  &  CIO, 
Caterpillar  Inc. 

Gary  Rossman,  Director,  Office 
of  Information  Management, 
Department  of  Public  Welfare, 
Commonwealth  of  Pennsylvania 

Ray  Thomas,  CEO,  Small  Business 
Unit,  Zurich  North  America 

Plus 

The  Super  Bowl 
Networking  Party 

Sunday  evening  is  for  the  Super 
Bowl.  Even  if  you're  not  a  football 
fan,  you  can  enjoy  the  networking— 
and  critiquingthe  new  crop  of  TV 
commercials  (were  they  really  worth 
all  that  money?). 

The  Gala  CIO 
Enterprise 
Value  Awards 
Ceremony  &  Dinner 

Tuesday  evening  we’ll  honor  all 
the  winners  in  the  industry  cate¬ 
gories— and  announce  the  winner 
of  the  Grand  CIO  Enterprise  Value 
Award— at  a  black-tie  reception, 
awards  ceremony  and  dinner.  It’s  a 
great  time  to  celebrate  with  your  CIO 
peers  as  we  close  out  the  conference. 


Business/IT  Case 
Study  &  Workgroups 
Robert  Austin, 
Associate  Professor  & 


Sponsored  by 
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HOW  TO  BE  A 


Learning  how  to 
accurately  interpret 
facial  expressions 

isn’t  easy,  but  it  can 
make  you  a  more 
effective  CIO 

BY  MERIDITH  LEVINSON 


"How  to  Be  a  Mind  Reader”  is  the  second  in 
an  occasional  series  titled  Advanced  Com¬ 
munications.  These  articles  will  feature  prac¬ 
tical  advice  designed  to  help  you  improve  the 
communication  skills  you  need  to  succeed 
as  managers  and  leaders.  For  more  on  this 
topic,  go  to  www.cio.com/communication. 


ERIC  GOLDFARB  KNOWS  that  tuning  into  body  lan¬ 
guage  and  facial  expressions  can  indicate  the  thoughts  and  feelings 
that  remain  unspoken.  He  also  knows  how  difficult  those  nonver¬ 
bal  cues  can  be  to  interpret.  During  a  budget  meeting  with  a  direct 
report  while  working  for  Global  Knowledge,  Goldfarb  noticed  that 
his  vice  president  kept  toying  with  her  necklace.  He  thought  this 
mannerism  was  an  indication  of  her  discomfort  with  the  financial 
target  he  was  proposing.  He  also  noticed  her  eyes  and  thought  they 
expressed  worry  over  the  budget  target.  He  repeatedly  asked  her 
during  the  meeting  if  she  thought  she  could  meet  the  budget,  and 
even  though  she  consistently  answered  yes,  Goldfarb  didn’t  believe 
her.  So  he  scheduled  a  follow-up  meeting  with  her  to  dig  deeper. 
She  ended  up  meeting  the  target  without  a  problem,  and  Goldfarb 
realized  that  he  wasted  his  and  her  precious  time  by  scheduling  the 
follow-up  meeting  and  by  dragging  out  the  first  one  with  repetitive 
questions.  What  could  Goldfarb  have  done  differently  to  more 
accurately  size  up  his  vice  president? 

Goldfarb,  now  the  CIO  of  auditing  firm  PRG-Schultz  Interna¬ 
tional,  was  astute  to  tune  into  her  body  language  and  facial 
expressions.  However,  because  body  language  can  be  misleading 
and  because  facial  expressions  can  be  hard  to  read  if  you’re  not  prac¬ 
ticed  at  it,  Goldfarb  needed  to  more  pointedly  probe  his  direct  report. 
Instead  of  continually  askingher,  “Areyou  comfortable?”  he  might 
have  said,  “It’s  really  important  for  me  to  have  your  buy-in  on  this 
target.  I  don’t  mean  to  pry  but  I  just  want  to  know  if  the  discom¬ 
fort  you  appear  to  be  showing  is  a  result  of  this  budget  target  or 
something  else.  If  it’s  the  target,  we  can  work  something  out.  ”  Had 
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COMPARE 


Advanced  Communications 


Goldfarb  taken  this  tack,  he  wouldn’t 
have  had  to  worry  that  his  incessant  ques¬ 
tioning  sent  a  message  to  this  individual — 
one  of  his  key  lieutenants — that  he  didn’t 
trust  her,  or  that  he  temporarily  lost  some 
credibility  in  her  eyes. 

Accurately  interpreting  the  meanings 
of  nonverbal  communications,  especially 


ity  can  make  CIOs  more  aware  of  unspo¬ 
ken  political  tensions  in  board  or  execu¬ 
tive  committee  meetings.  It  also  better 
equips  them  to  handle  sensitive  staffing 
situations  such  as  performance  reviews. 
Ekman  points  to  research  indicating  that 
managers  who  seem  responsive  to  the 
unspoken  emotions  of  their  staffs  are 


Behind  the  Smile 


COMPARE  AND  CONTRAST  the  expressions  below  on  the  face  of  Paul  Ekman, 
psychologist  and  author  of  Emotions  Revealed.  One  of  these  photos  shows  a  sin¬ 
cere  smile,  and  the  other  an  insincere  smile.  Which  is  which,  and  how  can  you  tell? 


ANSWER:  Photo  A  shows  an  insincere  smile.  Photo  B  shows  a  sincere  smile.  Note 
that  the  cheeks  are  higher  in  Photo  B  due  to  the  fact  that  in  a  sincere  smile,  you  use 
the  outer  part  of  the  muscle  that  orbits  the  eye. 


facial  expressions,  can  make  CIOs  more 
effective  leaders  and  managers,  says  Paul 
Ekman,  noted  psychologist  and  author 
of  Emotions  Revealed:  Recognizing 
Faces  and  Feelings  to  Improve  Commu¬ 
nication  and  Emotional  Eife.  Reading 
facial  expressions  is  a  particularly  useful 
skill  for  business  executives  because,  so 
often  in  business  settings,  people  don’t 
say  what  they  really  think.  If  CIOs  could 
recognize  how  different  emotions  mani¬ 
fest  themselves  on  the  face,  they’d  be  able 
to  discern  much  more  quickly,  for  exam¬ 
ple,  when  an  individual  is  starting  to  get 
angry.  They’d  also  be  able  to  identify 
when  people  are  trying  to  conceal  their 
emotions — such  as  fear,  contempt,  dis¬ 
gust  or  surprise.  This  knowledge  and  abil¬ 


more  successful  in  the  workplace  than 
managers  who  don’t. 

“  So  much  of  our  j  ob  [as  CIOs]  is  spent 
selling  things — ideas,  budgets,  influence. 
Becoming  sensitive  to  the  meanings  of 
facial  expressions,  while  tricky,  is  a  way  to 
find  out  very  quickly  who’s  allied  with  you 
and  who  might  be  angry  with  something 
you  said,”  says  Goldfarb. 

To  find  out  how  good  you  are  at  inter- 


More  About  Faces 

Facial  expressions  can  be  as  fleeting  as  a 
half-second,  even  though  they  are  the  clear¬ 
est  indicator  of  what  someone  is  feeling. 
Learn  how  to  catch  on  with  the  online 

Micro-Expression  Training  Tool.  Find  it  at 
www.cio.com/communication. 

cio.com 
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preting  facial  expressions,  try  our  quizzes, 
“How  Well  Can  You  Read  a  Face?”  on 
Page  76  and  online.  (The  quiz  online  is 
tougher  and  more  scientific,  partly  because 
it  gives  the  reader  a  very  short  time  to  read 
an  expression,  just  as  in  real  life.)  If  you 
want  to  know  whether  or  not  the  smile  the 
CEO  is  giving  you  is  sincere  or  whether  the 
CFO  is  contemptuous  of  you  when  you 
make  a  proposal,  keep  reading. 

THE  TRUTH  IN  FACIAL 
EXPRESSIONS 

While  facial  expressions  can  be  hard  to 
decipher  because  they’re  fleeting  (lasting 
anywhere  from  less  than  one-half  of  a  sec¬ 
ond  to  three  seconds)  and  because  people 
often  try  to  conceal  them,  they  are  in  fact 
the  clearest  indicator  of  what  someone  is 
feeling,  says  Ekman. 

“The  face  is  the  only  system  that  will 
tell  us  the  specific  emotion  that’s  occur¬ 
ring,”  he  says.  That’s  because  each  emo¬ 
tion  has  unique,  identifiable  signals  in  the 
face.  Emotions  manifest  themselves  in 
facial  expressions  because,  says  Ekman,  it 
became  useful  over  the  course  of  human 
evolution  to  let  others  know  when  we 
sense  danger.  Facial  expressions  have 
since  become  automatic.  Because  each 
emotion  has  unique  signals  in  the  face, 
facial  expressions  are  more  reliable  indi¬ 
cators  of  a  person’s  emotional  state  than 
body  language. 

Ekman  says  you  can  learn  the  funda¬ 
mentals  of  reading  facial  expressions  in 
about  an  hour  using  an  interactive  CD- 
ROM  he  has  put  together  that’s  available 
on  his  website,  www.paulekman.com. 
You  can  also  learn  to  read  facial  expres¬ 
sions  in  others  by  getting  to  know  how 
emotions  appear  on  your  own  face. 
Ekman  advises  individuals  to  look  in  a 
mirror  and  remember  a  personal  experi¬ 
ence  that  made  them  angry,  sad,  fearful  or 
disgusted  so  that  they  can  see  how  their 
expression  changes  as  the  emotion  washes 
over  them.  This  exercise  will  help  you  rec¬ 
ognize  muscle  movements  that  are  the 
clearest  indicators  of  a  particular  emotion. 


PHOTOGRAPHY  COURTESY  OF  PAUL  EKMAN 


Imagine  if  McAfee  could  protect  you  from  other  threats  the  way  it  protects  you  from  security  threats. 

You  never  know  when  security  threats  will  hit.  But  with  McAfee®  intrusion  prevention  technology,  you’ll  always  be  ready.  Our 
proactive  solutions  provide  real-time  protection,  so  you’re  free  to  focus  on  managing  your  business,  instead  of  constantly 
reacting  to  worms,  viruses,  and  hackers.  Even  better,  when  you  choose  McAfee,  you’re  protected  by  the  same  technology 
that  leading  Global  2000  companies  rely  on.  Discover  next-generation  security  today  at  proactive.mcafee.com 
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products.  All  other  registered  and  unregistered  trademarks  herein  are  the  sole  property  of  their  respective  owners.  ©  2004  Networks  Associates  Technology,  Inc.  All  rights  reserved. 
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Studying  the  photos  of  different  facial 
expressions  in  Emotions  Revealed  will 
further  help  you  learn  to  distinguish  the 
emotions.  Captions  under  each  photo 
describe  the  muscle  movements  in  the  face 
that  distinguish  a  sincere  smile  from  an 
insincere  smile,  and  that  signal  sadness, 


and  raised  upper  eyelids)  when  telling  a 
staff  member  that  she  did  not  get  a  pro¬ 
motion,  and  if  you  care  about  the  staff 
member  and  want  to  see  her  advance, 
Ekman  suggests  that  you  might  say  to  her, 
“I  know  that  was  bad  news  and  I  expect  it 
was  disappointing.  I  had  the  impression 


the  person  about  her  standing  in  the  com¬ 
pany  if  it’s  not  at  risk,  or  to  discuss  the 
areas  in  which  the  individual  needs  to 
improve. 

Ekman  says  that,  while  studying  facial 
expressions,  it’s  important  to  keep  in  mind 
that  they  do  not  reveal  what  is  generating 


How  Well  Can  You  Read  a  Face? 

Take  a  short  quiz  to  see  how  adept  you  are  at  readingfacial  expressions 

THIS  QUIZ  WILL  GIVE  you  a  taste  of  how  difficult  it  can  be  to  interpret  emotions  as  they  appear  on  a  face.  See  our  online 
quiz  at  www.cio.com/communication  for  a  tougher,  more  thorough  quiz.  Under  each  picture  are  three  different  emotions  the 
image  could  represent.  Pick  the  emotion  you  think  the  picture  communicates.  The  answers  are  on  the  bottom  of  the  page. 


O  0  ©  O  0 


A)  Surprise 

A)  Disgust 

A)  Disgust 

A)  Contempt 

A)  Fear 

B)  Fear 

B)  Fear 

B)  Contempt 

B)  Slight  happiness 

B)  Disgust 

C)  Anger 

C)  Surprise 

C)  Anger 

C)  Controlled  anger 

C)  Sadness 

SOURCE:  From  the  book  Emotions  Revealed:  Recognizing  Faces  and  Feelings  to  Improve  Communication  and  Emotional  Life,  by  Paul  Ekman. 


anger,  surprise,  fear,  contempt  or  disgust. 
By  studying  these  photos  and  captions, 
you’ll  learn  which  facial  movements  are  the 
clearest  indicators  of  a  particular  emotion. 
You’ll  also  learn  that  if  the  boss  doesn’t  con¬ 
tract  the  muscles  around  his  eyes  when  he 
smiles  at  you,  he’s  just  being  polite. 

USE  YOUR  KNOWLEDGE 

Once  you’ve  learned  to  automatically  and 
accurately  recognize  the  meanings  of  dif¬ 
ferent  facial  expressions,  you  can  decide 
whether  and  how  to  act  on  the  informa¬ 
tion  you  obtain  from  reading  faces. 

For  example,  if  you  pick  up  on  signs  of 
anger  (thinned  lips,  lowered  eyebrows, 


you  were  upset  and  wondered  if  it  would 
help  to  talk  about  it,”  or  simply,  “I  would 
be  glad  to  talk  to  you  now  or  at  a  later 
time  about  how  you  feel  about  it.” 
Ekman  cautions  against  asking  a  person 
in  this  situation  if  she  is  angry  because  it 
opens  the  CIO  up  to  an  attack. 

If  the  staff  member  shows  fear  (raised 
upper  eyelids,  tensed  lower  eyelids,  with 
eyebrows  raised  and  drawn  together), 
Ekman  says  her  expression  may  suggest 
that  she  is  concerned  about  her  future. 
Ekman  advises  supervisors  to  reassure 


QUIZ  ANSWERS: 


the  emotion,  only  that  the  emotion  is 
occurring.  Yet,  he  continues,  “If  we  are 
sensitive  to  the  expressions  of  another  per¬ 
son,  then  we  know  what  impact  we’re  hav¬ 
ing  on  them  and  what  emotion  they  might 
be  trying  to  conceal.”  In  other  words, 
we’re  a  lot  better  off  when  we  pay  atten¬ 
tion  to  and  know  how  to  assess  these  cues 
than  when  we’re  oblivious  to  them.  E0 


Share  your  stories  about  emotional  intelligence 
with  Senior  Writer  Meridith  Levinson.  She  can 
be  reached  at  mlevinson@cio.com. 


QUIZ  ANSWERS: 
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Dawninaofthe 

INTELLIGENT 
"SYSTEMS- 
ON-A-CH  I  P" 
EARNING 
IT  ACCOLADES 

BY  ELISABETH  HORWITT 


Broadband  IP  network  convergence  has  been  redefining  itself  of 
late,  both  as  a  market  and  as  a  corporate  strategy.  As  a  result, 
“We’ve  emerged  from  the  ‘kick-the-tires’  phase,”  says  John  Roese, 
CTO  of  Enterasys  Networks.  “CIOs  are  seeing  something  meaning¬ 
ful  at  the  end  of  the  tunnel  that  will  make  workers  more  efficient 
and  effective,”  and  their  companies  more  competitive. 


As  the  technology  has  matured,  and 
products  have  become  more  standards- 
based,  corporate  technical  decision 
makers  have  gained  confidence  that  a 
converged  IP  network  infrastructure 
will  provide  a  firm,  forward-migratable 
foundation  for  the  next  wave  of  strate¬ 
gic  business  applications.  “Our  cus¬ 
tomers  told  us  they  wouldn’t  go  with  IP 
telephony  if  they’re  just  getting  an  IP 
phone  that  acts  like  their  former  tele¬ 
phone,  or  if  they’re  just  moving  from 
one  proprietary  architecture  to  anoth¬ 
er,”  Roese  notes. 

A  lot  of  big  corporations  have  spent 
the  last  few  years  consolidating  data 
streams  from  multiple  applications 
and  access  points  onto  a  single  back¬ 
bone  based  on  TCP/IP  and  high-speed 
Ethernet.  Recent  statistics  tell  the 
story:  Dell’Oro  Group  expects  Gigabit 
Ethernet  port  shipments  to  at  least 
double,  from  13  million  units  in  2003 
to  more  than  27  million  in  2004, 
according  to  Seamus  Crehan,  a  direc¬ 
tor  at  the  networking  and  telecommu¬ 


nications  research  firm. 

More  recently,  companies  have 
begun  moving  beyond  broadband  con¬ 
verged  data  networking  toward  a  fully 
integrated,  enterprise  network  infra¬ 
structure  that  can  handle  the  full  range 
of  services:  from  teleconferencing  to 
storage  networking  to  server  clustering. 
Helping  them  to  reach  their  goal  with  a 
minimum  of  pain  is  an  emerging  breed 
of  intelligent  network  hardware  based 
on  highly  integrated  semiconductors 
known  as  “systems-on-a-chip.” 

“Whether  you’re  talking  about  VoIP 
[voice  over  IP]  or  iSCSI  storage,  it’s  not 
just  about  high-speed  connectivity  any¬ 
more;  you  have  to  move  bits  around  the 
network  intelligently,  in  order  to  ensure 
security,  reliability,  and  quality  of  serv¬ 
ice,”  says  Roese. 

Increasingly  intelligent  and  versatile 
IP  network  infrastructures  are  forming 
the  basis  for  a  wave  of  new  applications 
that  are  providing  businesses  with 
strategic  advantages  as  well  as  signifi¬ 
cant  cost  savings. 


Moving  Beyond  VoIP 

Consider,  for  example,  the  veritable 
explosion  in  American  companies’  use 
of  VoIP  technology  that  merges  voice 
and  data  onto  a  single  IP/Ethernet 
infrastructure.  A  2003  IDC  report  pre¬ 
dicted  that  U.S.  spending  on  hosted 
VoIP  will  reach  approximately  $281 
million  this  year,  growing  to  $1  billion 
next  year  (2005).  By  2007,  revenue  is 
expected  to  reach  $6.7  billion. 

In  a  2003  Nemertes  Research  survey 
of  42  large  corporations,  62%  of  respon¬ 
dents  said  they  were  currently  using  IP 
telephony;  19%  were  running  a  trial; 
and  the  rest  planned  to  implement  it 
within  the  next  year  or  two. 

The  reported  return  on  investment 
(ROI)  and  measurable  benefits  the  VoIP 
pioneers  cite  are  impressive,  and 
include  lower  maintenance  and  man¬ 
agement  costs,  the  result  of  dealing  with 
one  set  of  network  boxes,  interfaces, 
and  cables  instead  of  two  or  more. 
Companies  report  saving  thousands  of 
dollars  per  month  by  using  IP  telephony 
to  bypass  long-distance  toll  calls;  global 
firms  have  saved  much  more  on  inter¬ 
national  calls. 

And  cost  savings  are  just  the  begin¬ 
ning.  Converged  IP  networking  has 
paved  the  way  for  a  new  generation  of 
applications  and  software  tools  that  are 
boosting  end-user  productivity  and 
helping  companies  gain  a  competitive 
advantage.  »» 


DAWNING  OF  THE  CONVERGED  IP  NETWORK 
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Worldwide  Ethernet  ASSP/ASIC  Semiconductor 
Revenue  Share  by  Vendor,  2003 

Other  (11.4%) 


Realtek 
(8.6%) 

LSI  Logic 
(9.0%) 

Agere 
(9.1%) 

Marvell  (14.0%J 


Broadcom 

(25.8%) 


Intel 

(22.1%) 


Total  =  $2.07B 

Note:  "Other"  includes  ADMtek  (now  Infineon),  Agilent,  AMCC, 
AMD,  Cicada  (now  Vitesse),  Fujitsu,  IBM  Micro,  NEC, 
PMC-Sierra,  SwitchCore,  Texas  Instruments,  Toshiba, 

Vitesse,  and  Zarlink.  SOURCE:  IDC,  2004 


Instant  voice  messaging  is 
a  perfect  example,  says  Jeff 
Snyder,  a  research  vice  presi¬ 
dent  at  Gartner.  “You  click  on 
my  name,  and  my  phone 
rings,  no  matter  where  I  am; 
as  long  as  I’m  logged  onto  the 
network,  it  knows  how  to 
reach  me.  Drag  in  a  col¬ 
league’s  name  and  you  have 
an  ad  hoc  conference  call.” 

Another  promising  devel¬ 
opment  is  convergence  on 
end-user  devices  across  a 
unified  end-to-end  infra¬ 
structure.  Broadcom  has  introduced 
switch  and  VoIP  chipsets,  for  example, 
that  will  enable  OEMs  to  build  Wi-Fi®- 
enabled  cell  phones,  “so  you  can  roam 
between  the  cellular  and  Wi-Fi  IP  teleph¬ 
ony  worlds  with  one  device,”  says  Roese. 

User  Demand  Driving 
IP  Convergence 

Indeed,  user  demand  is  one  of  the  biggest 
drivers  behind  converged  IP  networking 
strategies.  “Users  want  voice,  data,  and 
video  services  delivered  in  an  integrated 
fashion,”  to  support  the  growing  body  of 
multimedia  productivity  tools  and  appli  ¬ 
cations  being  offered  on  desktops,  lap¬ 
tops,  and  the  latest  mobile  computing 
devices,  says  Charles  Salameh,  vice  presi¬ 
dent  of  emerging  solutions  at  Bell 
Canada,  a  telecom  service  provider  and 
systems  integrator. 

IT  decision  makers  see  converged  IP 
networking  as  their  best  shot  at  meeting 
users’  multimedia  needs  over  the  long 
haul.  And  they’re  counting  on  intelligent 
Gigabit  Ethernet  hardware  to  help  them 
with  quality  of  service  (QoS)  and  securi¬ 
ty  management,  across  an  increasingly 
diverse  set  of  network  services  and 
media  traffic. 

To  address  this  demand,  networking 
vendors  are  providing  their  latest  Gigabit 
Ethernet  products  with  the  intelligence  to 
handle  these  tasks  and  a  great  deal  more. 
And  much  of  this  high-level  intelligence 
resides  in  silicon  solutions  that  have  tra¬ 
ditionally  been  the  “nuts  and  bolts”  of 
networking  equipment,  says  Kevin  Tolly, 
president  of  The  Tolly  Group  research 
firm.  In  the  last  year  or  two,  chipmakers 
like  Broadcom  “have  gone  up  the  stack 
tremendously,  and  have  preintegrated  a 
lot  of  the  functionality  and  capabilities,” 
he  adds. 

“The  trend  is  toward  supporting  a 


higher  capacity  of  networking  interfaces 
and  a  broader  range  of  services  on  the 
same  piece  of  equipment,  and  using  sili¬ 
con  intelligence  to  handle  sophisticated 
management  of  converged  applications 
running  across  a  single  Ethernet  net¬ 
work,”  says  Ford  G.  Tamer,  group  vice 
president  of  Broadcom’s  Network 
Infrastructure  Business  Group.  “Such 
migration  to  converged  IP  networks  can 
mean  huge  savings  in  maintenance 
costs,”  he  adds.  “You’re  managing  your 
desktops,  laptops,  VoIP  phones,  wireless 
access  points,  security,  and  storage  needs 
within  the  same  equipment,  instead  of 
across  two  or  more  boxes.” 

Broadcom  in  Step 
with  Key  Trends 

Broadcom®  silicon  solutions  are  at  the 
forefront  of  these  trends.  For  example, 
systems  using  Broadcom’s  StrataXGS® 
family  of  enterprise  switch  products  offer 
enhanced  scalability  and  multilayer  intel¬ 
ligence  to  address  the  different  service- 
level  needs  of  voice,  video,  and  data  traf¬ 
fic  within  business  environments,  Tamer 
says.  This  enables  IT  managers  to  maxi¬ 
mize  network  uptime  and  ensure  securi¬ 
ty  while  promoting  ease  of  management. 

Furthermore,  Broadcom’s  NetXtreme™ 
II  C-NIC  (Converged  NIC)  is  the  first 
Gigabit  Ethernet  NIC  that  can  simultane¬ 
ously  perform  storage  networking,  high- 
performance  clustering,  accelerated  data 
networking,  and  remote  system  manage¬ 
ment  on  a  standard  Ethernet  network, 
according  to  Allen  Light,  Broadcom’s  serv¬ 
er  controller  product  line  manager. 

One  of  the  big  advantages  of  the  C-NIC 
is  its  ability  to  offload  a  variety  of  func¬ 
tions  from  the  host  CPU,  “freeing  up  pro¬ 
cessing  cycles  and  enabling  the  server  to 
handle  other  functions,”  says  Sean  Lavey, 
a  program  manager  at  IDC,  For  example, 


Broadcom’s  C-NIC  has  a  TCP/IP  Offload 
Engine  (TOE)  that  takes  over  the  process¬ 
ing  of  TCP/IP  stacks. 

During  a  Microsoft®  benchmark  test, 
Broadcom’s  BCM5706  C-NIC,  equipped 
with  a  TOE,  had  a  throughput  of  1.8 
Gbit/sec  at  20%  CPU  utilization,  and  used 
only  3  watts  per  Gigabit.  In  comparison, 
the  same  Windows®  server  equipped 
with  an  Intel®  non-TOE  Gigabit  Ethernet 
NIC  had  about  1.5  Gbit/sec  at  95%  CPU 
utilization,  and  used  90  watts  per  Gigabit. 

Other  key  C-NIC  offloading  capabili¬ 
ties  include: 

■  Remote  Direct  Memory  Access 
(RDM A)  that  enables  servers  in  high¬ 
speed  clusters  to  write  directly  to  each 
other’s  memory,  bypassing  the  bus. 
This  eliminates  potential  bus  bottle¬ 
necks  and  boosts  performance, 
enabling  clustered  X-86-based  servers 
to  handle  applications  that  ordinarily 
require  high-end,  and  much  more 
expensive,  Unix  machines. 

■  Support  of  the  iSCSI  protocol  that 
allows  IT  professionals  to  move  block- 
level  storage  from  a  dedicated  Fibre 
Channel  SAN  onto  a  shared  converged 
IP  network. 

C-NICs,  and  other  recent  Broadcom 
offerings,  also  represent  the  leading  edge 
of  an  important  industry  trend,  in  which 
network  equipment  manufacturers  build 
their  products  with  more  “systems-on-a- 
chip”  and  fewer  custom  ASICs.  In  this 
way,  they  can  bring  new  products  to  mar¬ 
ket  faster  and  more  cheaply,  which  trans¬ 
lates  into  more  choices  and  lower  costs 
for  customers,  says  IDC’s  Lavey. 

Enterasys  used  Broadcom’s  StrataXGS 
silicon  as  a  critical  component  for  its 
Matrix  C2  line  of  layer  3  stackable  IP 
switches,  announced  in  September  2004. 
“The  intelligence  of  Broadcom’s  compo¬ 
nentry  provides  the  foundation  for  many 
security  functions,  enabling  support  for 
multimodal  systems,  predictability,  and 
integration,”  says  Roese.  By  using 
Broadcom  silicon,  with  built-in  capabili¬ 
ties  like  classification  and  policing, 
Enterasys  was  able  to  deliver  the  new 
switches  in  about  eight  months,  “more 
cheaply,  and  compromise-free,”  he  adds. 
In  contrast,  building  the  product  from 
scratch  internally,  using  custom  ASICs, 
would  have  taken  about  18  months.  The 
benefit  for  Enterasys’  customers:  “They 
get  state-of-the  art  equipment,  while  still 
keeping  up  with  the  market’s  progression 
toward  faster,  cheaper  equipment.” 


DAWNING  OF  THE  CONVERqED  IP  NETWORK 


ADVERTISING  SUPPLEMENT 


IP  CONVERGENCE: 

It's  All  About  the  ROI 


Figuring  out  the  potential  paybacks  and  drawbacks 
of  a  hot  new  technology  can  be  tricky  enough,  but  doing  it  for 
an  enterprisewide  strategy  like  IP  networking  convergence  is 
particularly  challenging. 

For  example,  corporate  decision  makers  have  to  weigh  priorities 
against  other  capital  investments,  and  take  into  account  equipment 
depreciation  life  cycles. 

A  recent  survey  of  42  large  companies  by  Nemertes  Research 
found  that  IT  executives  are  casting  a  wide  net  when  it  comes  to 
potential  return  on  investment  (ROI)  of  IP  converged  networking. 
When  asked  what  their  goals  were  for  implementing  IP  telephony, 
for  instance,  26.3%  of  respondents  said  centralized  management; 
23.7%  cited  future  proofing;  23.7%  cited  moves,  adds,  changes 
(MAC)  cost  reduction;  and  23.7%  said  applications  and  features. 

Later  this  year,  Broadcom  and  its  OEM  partners  will  begin  rolling 
out  a  new  generation  of  intelligent  Gigabit  Ethernet  equipment  that 
will  make  broadband  convergence  even  easier  to  justify. 

To  take  just  one  example,  Hewlett-Packard  and  Broadcom  are 
working  on  a  Gigabit  Ethernet  adapter  for  the  HP  Proliant  server  that 
will  consolidate  "lights  out"  management  ports  and  IP  storage  in  the 
native  network  connections,"  says  Paul  Perez,  vice  president  of 
storage,  networks,  and  infrastructure  at  Hewlett-Packard's  Industry 
Standard  Server  business.  "The  performance  will  be  the  equivalent 
of  a  cluster  interconnect,  but  over  standard  Ethernet,"  he  says. 


Taking  into  account  administrative  overhead  and  "all  the  other  cost 
burdens  of  connecting  the  server  to  the  switch  through  a  data  cen¬ 
ter  infrastructure,"  the  converged  Gigabit  Ethernet  adapter  could 
save  a  company  hundreds  of  dollars  per  port  in  total  cost  of  owner¬ 
ship,  Perez  says. 

In  for  the  Long  Haul 

Above  all,  what's  driving  many  companies  toward  converged  IP 
networking  is  a  growing  perception  that  the  technology  is  here  for 
the  long  haul,  says  John  Roese,  CTO  of  Enterasys  Networks.  "Our 
customers  tell  us  that  they  wouldn't  be  going  forward  with  [con¬ 
verged  networking  deployments]  if  they  didn't  see  it  as  the  founda¬ 
tion  for  the  next  wave  of  strategic  applications." 

Business  decision  makers  need  to  think  long  term  as  well  as 
short  term  when  it  comes  to  ROI,  says  Jeff  Snyder,  a  research  vice 
president  at  Gartner.  "Companies  converting  to  a  converged  net¬ 
work  today  are  doing  so  on  the  assumption  that  there  will  be  meas¬ 
urable  value  going  forward,"  he  says.  "And  there  will  be." 

As  the  Nemertes  survey  shows,  senior  executives  see  future 
proofing  as  a  strong  justification  for  getting  started  with  IP  net¬ 
working  convergence.  By  setting  up  the  basic  infrastructure,  getting 
technical  staff  up  to  speed,  and  testing  the  waters  with  a  limited 
deployment,  a  company  can  position  itself  to  beat  competitors  to 
the  big  paybacks  down  the  road. 


Not  All  Silicon  Is  Created  Equal 

All  this  is  good  news  for  corporate  decision 
makers.  But  Tolly  adds  a  caveat:  It’s  becom¬ 
ing  crucial  that  IT  executives  “look  under 
the  hood”  when  shopping  for  Gigabit 
Ethernet  equipment,  and  ask  a  few  pointed 
questions  about  the  brand,  feature  sets,  and 
configuration  of  the  underlying  chipsets — 
before  making  a  buying  decision.  One 
important  question  to  ask:  Will  it  interop¬ 
erate  with  other  types  of  equipment? 

Broadcom  has  a  three-pronged  interop¬ 
erability  strategy.  First,  it  has  been  an 
industry  leader  in  both  the  development 
and  implementation  of  key  industry  stan¬ 
dards  like  RDMA,  iSCSI,  Wi-Fi,  and  the 
IEEE  802. lx  Extensible  Authentication 
Protocol.  Second,  it  does  extensive  testing 
to  ensure  all  of  its  products  are  backward 
and  forward  interoperable,  and  can  sup¬ 
port  legacy  applications.  Finally, 
Broadcom  provides  support  for  vendor- 
and  operating  system-specific  de  facto 
standards  that  enable  networking  equip¬ 
ment  to  work  seamlessly  with  the  host 
computing  platform. 


For  example,  Broadcom  currently  pro¬ 
vides  the  only  TOE  on  a  chip  that  interfaces 
with  Microsoft’s  TCP  Chimney.  Support  of 
Chimney  enables  TCP/IP  Offload  Engines 
to  interface  directly  with  Windows,  offload¬ 
ing  TCP/IP  processing  for  all  network  traf¬ 
fic,  Broadcom’s  Light  explains.  Without  that 
operating  system  support,  a  TOE  can  only 
work  on  an  iSCSI  adapter  that  bypasses  the 
operating  system. 

How  Well  Does  It  Perform? 

IT  managers  also  need  to  check  published 
benchmark  tests  that  compare  the  per¬ 
formance  of  the  same  type  of  chip  from 
different  vendors.  eTesting  Laboratories 
found  that  Broadcom’s  NetXtreme  64-bit 
PCI-X  Gigabit  Ethernet  server  adapter 
achieved  significantly  higher  throughput 
rates  compared  with  the  Intel  Pro  1000  XT 
server  adapter. 

Network  equipment  performance 
depends  not  only  on  the  power  of  the 
underlying  silicon,  but  on  “architectural 
choices  that  chipmakers  are  increasingly 
making,”  says  Tolly.  Customers  should  ask 


questions  like:  “Do  they  support  the  latest 
industry  standards?  How  much  memory 
or  intelligence  did  they  put  on  the  chip? 
Are  tasks  like  IPSec  encryption  handled 
by  the  main  processor  or  offloaded  to  a 
subsidiary?” 

The  bottom  line,  according  to  Tolly: 
There  is  a  big  difference  between  custom 
ASICs  and  standards-based,  system-level 
semiconductors  designed  and  tested  to 
work  together  across  the  network.  “You 
can  do  enough  research  on  a  chipmakers 
products,  like  Broadcom’s,  to  get  a  pretty 
good  idea  of  what  those  silicon  solutions 
can  do  for  you,”  says  Tolly.  “Then  you  can 
ensure  good  performance  across  your  net¬ 
work  by  ensuring  that  all  your  equipment 
is  built  on  that  chipmakers  hardware.”  ■ 

Broadcom®,  StrataXGS®,  and  NetXtreme™  are 
trademarks  of  Broadcom  Corporation  and/or  its 
affiliates  in  the  United  States  and  certain  other 
countries.  Wi-Fi®  is  a  trademark  of  the  Wi-Fi 
Alliance.  Microsoft®  and  Windows®  are  trade¬ 
marks  of  Microsoft  Corporation.  Intel®  is  a  trade¬ 
mark  of  Intel  Corporation.  Any  other  trademarks 
or  trade  names  mentioned  are  the  property  of 
their  respective  owners. 
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Download  our  new  white  paper,  “Critical  Steps 
for  Successful  VoIP  Deployment,”  today  at 

www.gobroadcom.com/VolP 


LOOK  WITHIN  — 

YOUR  NETWORK  IS  MORE 
POWERFUL  THAN  YOU  REALIZE 

To  engage  the  immense  power  of  IP  convergence  —  data,  voice,  applications,  networks  —  the  underlying  technology 
must  function  securely,  reliably  and  efficiently.  Only  Broadcom®  offers  end-to-end  networking  chips  with  the  built-in 
power  to  manage  convergence  —  from  core  storage  and  servers,  through  switch  fabrics  to  desktops,  multi-functional 
devices  and  beyond.  Broadcom’s  wired  and  wireless  technology  throughout  the  network  reduces  complexity  and 
enhances  the  end-user  experience.  With  Broadcom,  there  is  no  gap  between  legacy  and  future  technologies  —  only 
seamless  communication  at  maximum  speed.  Technology  advances.  Networks  expand.  Control  is  maintained. 


Broadcom,'  the  pulse  logo,  Connecting  everything,''  and  the  Connecting  everything  logo  are  trademarks  of  dtaadcom  Corporation  and/or  its  subsidiaries  in  the  United  States  and  certain  other  countries.  All  other  trademarks  are  the  property  of  their  respective  owners. 
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takeaways— people  I 
would  not  have  met 
otherwise.  Diverse, 
professional,  interesting, 
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Authority 
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dancing  the  Art  &  Science  of  Security 


iy's  security  executives  are  required  to 
Drm  difficult  and  constant  balancing  acts 
een  the  art  and  science  of  security,  con- 
)usly  weighed  against  the  needs  of  the 
ness.  Getting  the  “science”  part  of  the 
ition  right  is  the  easier  part.  The  tech- 
gies  are  known  entities,  and  better  ones 
jinue  to  evolve.  There  are  quantitative 
surements  around  such  issues  as  intru- 
detection,  forensics  and  regulatory 
ipliance,  along  with  more  mature 
jnpts  to  quantify  the  ROI  of  security, 
he  “art”  of  security  that’s  the  harder 
(-the  art  of  diplomacy,  of  persuasion, 
:tting  into  and  understanding  other 
Jsets.  It’s  everything  from  establishing 


security  procedures  everyone  will  actually 
follow.  It's  fostering  positive  relations  with 
senior  executives  and  the  board  of  directors. 
It’s  getting  the  staff  to  think  like  a  hacker  or 
terrorist  to  get  ahead  of  potential  threats. 

We’ll  examine  this  complex  balancing  act  by 
looking  at  what  the  top  practitioners  are 
thinking  and  doing,  and  by  listening  to  what 
leading  security  and  privacy  experts  think 
will  affect  the  landscape  of  the  future. 

For  more  information 

call  800.366.0246  or  visit 
www.csoonline.com/conferences 
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Change 

Catalog  retailer  Lillian  Vernon 
learned  the  hard  way  how  critical 
change  management  can  be 

to  a  turnaround 

BY  LAUREN  GIBBONS  PAUL 

JONATHAN  SHAPIRO  REMEMBERS  THE  INSTANT  HE 
knew  he  wanted  to  hire  Tom  Scott  as  CIO  of  Lillian  Vernon.  In  Scott’s 
first  interview  at  the  Rye,  N.Y.,  headquarters  of  the  specialty  catalog 
retailer  in  August  2003,  he  wowed  company  president  Shapiro  when 
he  responded  to  a  routine  question  about  a  customer  database  he  had 
been  working  on  at  his  then  current  employer. 

Shapiro  asked  Scott  what  kind  of  infrastructure  he  was  planning 
to  deploy  the  database  on.  “I’ll  never  forget  what  Tom  said,”  Shapiro 
says.  ‘“I  knew  I  could  build  a  state-of-the-art  system  that  would  deliver 
everything  the  business  needed,  but  I  knew  I  couldn’t  justify  it.’” 
Shapiro’s  heart  started  beating  faster.  In  25  years  of  working  with 
CIOs,  he  had  never  encountered  one  so  intently  focused  on  business 
needs.  “I  said  to  myself,  This  is  our  guy,”  he  recalls. 

It  would  take  a  special  guy  indeed  to  head  up  a  sweeping  transfor¬ 
mation  of  Lillian  Vernon’s  entire  technical  architecture.  Shapiro 
was  part  of  the  group  of  investors,  including  ZelnickMedia  and 
Ripplewood  Holdings,  that  had  just  purchased  Lillian  Vernon  for 
the  fire  sale  price  of  $60.5  million.  The  53-year-old  mail-order  com¬ 
pany  then  became  known  as 
Direct  Holdings  Worldwide. 

Following  three  straight  years 
of  precipitously  declining  rev¬ 
enue  and  stock  value,  the  new 
owners  took  the  company  pri¬ 
vate.  In  fiscal  year  2003,  losses 
topped  $18  million. 

To  stop  the  bleeding,  Shapiro 
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►  How  a  midmarket 
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ambitious  IT  overhaul 
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project  management 
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Change  Management 


and  his  partners  aimed  to  beef  up  sales  by 
freshening  the  product  line,  adding  a  fleet  of 
sales  representatives  to  sell  products  from  their 
homes  and  improving  the  customer  shopping 
experience  in  every  channel — be  it  catalog, 
Web,  outlet  store  or  one  of  the  new  reps.  To 
do  this,  management  decided  to  completely 
overhaul  the  archaic  systems  on  which  the 
company  grew  up.  In  sharp  contrast  to  the 
conservative  approach  taken  by  most  midsize 
enterprises,  the  management  team  decided, 
fatefully,  that  speed  was  crucial.  The  ambi¬ 
tious  plan  called  for  the  installation  of  10 
major  applications  at  the  same  time,  with  a 
tight-but-not-impossible  budget  of  $7  million 
and  the  end  of  2004  as  the  finish  line. 

Scott,  who  had  worked  as  COO  at  cata¬ 
log  retailer  Coldwater  Creek,  was  hired  as 
executive  vice  president  of  operations  and 
CIO.  He  began  working  out  of  Lillian  Ver¬ 
non’s  office  in  Virginia  Beach,  Va.,  in  Sep¬ 
tember  2003.  Given  the  scope  of  the  IT 
projects  and  the  company’s  limited  resources, 
Shapiro  agreed  to  lead  the  IT  implementa¬ 
tion  team  along  with  Scott — a  team  that 
failed  to  plan  for  the  effects  of  so  much 
change  so  fast.  Employees  resisted  mightily, 
avoiding  training  and  blaming  new  applica¬ 
tions  for  their  frustration.  The  company  will 
now  miss  its  ambitious  time  frame  for  rolling 
out  all  of  its  new  applications  and  will  have  to 
wait  until  next  year’s  holiday  season  to  cap¬ 
italize  on  new  efficiencies  wrought  by  IT.  Yet 
as  serious  as  their  missteps  were,  both  Scott 
and  Shapiro  have  managed  to  incorporate 


the  lessons  learned  into  the  continuing  roll¬ 
outs.  That’s  proved  critical  because  the 
deployment  of  the  most  important  applica¬ 
tion — the  order-entry,  order  management 
and  warehouse  management  system — is 
only  one-third  finished.  Both  men  acknowl¬ 
edge  in  hindsight  that  the  plan  was  too  ambi¬ 
tious.  “In  a  world  of  perfect  information,  we 
might  have  chosen  to  schedule  this  differ¬ 
ently,”  says  Shapiro.  “The  business  side  had 
the  plans  in  place  before  I  came  on  board,” 
adds  Scott.  “I  doubt  they  understood  at  the 
time  how  much  they  were  taking  on.” 

In  a  way,  that  can-do  attitude  Scott 
observed  in  his  peers  was  what  attracted  him 
to  the  job.  “Of  all  the  places  I’ve  been  and  all 
the  projects  I’ve  been  involved  in,  this  is  the 
most  aggressive  concurrent  set  of  projects  I’d 
ever  seen,”  he  says.  He  was  particularly 
struck  by  the  business  managers’  apparent 
readiness  to  take  on  so  much  change  at 
once — something  he  perhaps  should  have 
questioned. 

Everything  Must  Go 

Before  the  ink  was  dry  on  the  agreement  to 
buy  Lillian  Vernon,  Strauss  Zelnick  (chairman 
and  CEO  of  Direct  Holdings)  and  Shapiro 
were  convinced  they  had  to  rip  out  all  of  the 
company’s  systems  and  start  over  with  a  blank 
slate.  For  one  thing,  most  of  the  applications 
were  homegrown,  mainframe-based  systems 
that  were  15  or  more  years  old  and  totally 
lacking  in  capabilities  that  any  modern  cata- 
loger  would  consider  essential. 


For  example,  since  the  order-entry  and  cus¬ 
tomer  service  systems  were  not  Web-enabled, 
call  center  agents  could  not  surf  the  website 
along  with  people  who  called  in  to  place  an 
order.  This  frustrated  and  embarrassed  the 
agents  and  baffled  the  customers.  Lillian  Ver¬ 
non  was  largely  in  the  business  of  selling  gifts, 
such  as  personalized  baby  blankets  and 
engraved  charm  bracelets.  But  the  company 
could  not  even  offer  gift  wrap  due  to  the 
inflexibility  of  the  customer  service  system. 

Merchandise  buyers’  efforts  to  dig  down 
into  the  details  of  which  items  were  selling  in 
what  colors,  sizes  and  styles — the  bread  and 
butter  of  any  retail  operation — were  ham¬ 
strung  because  Lillian  Vernon  had  to  pay  its 
customer  database  outsourcer  for  each  indi¬ 
vidual  inquiry.  It  was  cost-prohibitive  to  ana¬ 
lyze  the  trends  behind  the  sales  data,  and 
without  that  knowledge,  the  product  mix  had 
grown  stale  and  unappealing. 

Another  shortcoming:  Lillian  Vernon  did 
not  have  a  digital  asset  management  (DAM) 
system.  So  the  buyers  and  creative  services 
people  who  put  the  catalog  and  Web  store 
together  had  to  manage  product  photos  and 
descriptions  manually,  an  extremely  ineffi¬ 
cient  and  mistake-prone  process. 

Among  the  applications  Shapiro  and  Scott 
targeted  for  replacement  were  the  order-entry, 
customer  service  and  warehouse  management 
systems.  Also  on  the  shopping  list:  the  cus¬ 
tomer  database,  a  DAM  system,  a  merchan¬ 
dise  planning  application,  an  overhaul  of  the 
website  and  a  new  financial  module. 
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Change  Management 


The  business  acumen  that  Shapiro  admired 
in  Scott  came  through  when  he  and  his  fellow 
executives  began  to  select  the  platforms  on 
which  Lillian  Vernon  would  rebuild  its  busi¬ 
ness.  There  was  no  one  product  or  platform 
that  would  perform  all  of  the  functions  the 
company  needed,  so  integrating  systems  from 
a  variety  of  vendors  was  pretty  much  a  given. 
But  for  a  company  of  its  size,  best-of-breed 
was  not  an  option.  “We  are  integrating  third- 
party  software  that  best  meets  our  budget  and 
our  functional  needs.  We  wanted  the  best  rea¬ 
sonable  fit  for  both,  ”  says  Scott.  If  that  meant 
cutting  a  few  corners  here  and  there,  settling 
on  a  system  that  might  not  have  been  ideal  for 
financial  reasons,  so  be  it.  The  executive  team 
approved.  “Tom  is  extraordinarily  good  at 
mixing  creative  technology  solutions  with 
cost-effective  solutions,”  says  Shapiro. 

The  most  important  piece  of  technology — 
from  a  business-critical  perspective — was  the 
system  for  order  entry,  customer  service  and 
warehouse  management.  The  company’s  new 
direct-sales  channel,  due  to  be  up  and  running 
in  early  2005 — requires  more  flexible  cus¬ 
tomer  service  technology.  In  the  new  model, 
sales  reps  host  parties  associated  with  holidays 
throughout  the  year — a  Christmas  ornament 
swap  in  December,  for  example.  Unlike  the 
old  Avon  model,  orders  will  be  shipped 
directly  to  customers’  homes. 

“We’ll  have  reps  calling  in  orders  going  to 
multiple  different  customer  addresses.  The 
reps  will  have  their  choice  of  calling  in  orders 
to  the  call  center  or  placing  them  on  the  Web. 
Our  old  system  was  based  on  single-customer 
orders,”  says  Scott.  The  system  would  also 
need  to  handle  the  complexity  of  a  multi¬ 
tiered  commission  structure,  in  which  reps 
who  recruit  other  successful  reps  will  be  eligi¬ 
ble  for  higher  commissions  than  those  who 
just  host  parties.  Scott  created  a  vendor  short¬ 
list  and  was  in  the  midst  of  the  usual  selection 
process  when  he  was  thrown  a  curveball. 

In  January  2004,  Direct  Holdings  an¬ 
nounced  its  acquisition  of  Time  Life,  a  pur¬ 
veyor  of  music  and  videos.  Still  in  the  process 
of  selecting  the  new  systems,  Scott  had  to  drop 
everything  and  merge  Time  Life’s  call  center, 
data  center  and  warehouse  into  the  Lillian 


licenses,  Lillian  Vernon  could  reduce  the  cost 
of  its  most  expensive  system  by  about  40  per¬ 
cent.  “It  was  too  good  of  a  deal  to  pass  up,” 
says  Scott.  The  first-year  cash  outlay  for  Ecom- 
etry  came  in  at  $2. 1  million — by  far  the  biggest 
item  on  the  shopping  list  (see  “IT  Shopping 
Spree,”  Page  86),  but  far  less  than  it  would  have 
cost  to  get  similar  applications  from  Commer- 
cialWare,  another  vendor  on  his  shortlist. 

Lillian  Vernon  would  also  install  its  first 
DAM  system  to  handle  product  images  and 
descriptions  for  the  catalog  and  website.  The 


Vernon  facility.  He  did  this  in  a  speedy  75  busi¬ 
ness  days,  after  which  he  refocused  on  the  ini¬ 
tial  changes  at  Lillian  Vernon. 

Always  on  the  lookout  for  a  bargain,  Scott 
discovered  that  Time  Life  had  excess  licenses 
for  its  order-entry  and  management  system 
from  Ecometry.  That  was  a  stroke  of  luck,  since 
Ecometry  was  on  his  shortlist  of  order-entry 
and  customer  service  vendors.  Although  Scott 
entertained  a  few  doubts  that  the  product  was 
the  best  from  a  pure  technology  perspective,  he 
soon  put  them  aside.  By  leveraging  the  unused 


When  CIO  Tom  Scott 

began  work  on  the 
transformation,  change 
management  did  not 
figure  in  his  list  of 
potential  disasters. 
With  so  much  work 
ahead,  the  softer  side 
wasn’t  on  his  early 
radar  screen. 
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than  $1  million  over  the  next  three  years.  Find  out  more  about  how  we’re  helping 
Daybreak  Venture  connect  the  dots  at  sbc.com/dots.  GOING  BEYOND  THE  CALL? 
egistered  trademarks  o!  SBC  Knowledge  Ventures,  L.P.  and/or  its  affiliates.  ©2004  SBC  Knowledge  Ventures.  L.P.  AS  rights  reserved. 
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technology  selection  team  chose  Lago  from 
Comosoft,  a  provider  of  advertising  media 
management  and  production  software. 

Merchandise  planning  and  inventory  con¬ 
trol  was  another  critical  area,  since  a  fusty 
product  line  was  largely  to  blame  for  the  com¬ 
pany’s  post-2000  slide.  “We  needed  a  better 
understanding  of  what  was  selling  and  in  what 
colors  and  quantities,”  says  Scott.  Evant  Mer¬ 
chandise  Planning  from  Evant,  a  developer  of 
merchandising  software,  got  the  nod. 

Lillian  Vernon’s  website  was  due  for  an 
overhaul,  as  well.  On  the  previous  site,  users 
couldn’t  track  orders  across  channels.  (For 
example,  if  you  place  an  order  over  the  phone, 
you  can  track  it  only  by  calling.)  And  the  site 
was  not  robust  enough  to  handle  the  leap  in 
traf  fic  that  would  come  along  with  the  growth 
company  executives  are  hoping  to  see.  The 
team  tapped  Fry,  a  provider  of  e-commerce 
systems  and  services,  to  do  that  work,  which 
is  now  complete.  However,  in  anticipation  of 
the  holiday  season  (which  begins  with  Hal¬ 
loween  and  accounts  for  60  percent  of  annual 
revenue),  Lillian  Vernon  stops  all  new  systems 
deployments  on  Oct.  1  of  each  year.  The  com¬ 
pany  therefore  is  holding  off  on  the  launch  of 
the  new  site  until  January. 

Shapiro  admits  it  is  frustrating  waiting  for 
the  new  site  to  launch.  “It’s  a  little  disappoint¬ 
ing.  We  would  definitely  deploy  it  now  if  our 
business  was  the  same  month  over  month,”  he 
says.  But  with  such  a  large  chunk  of  revenue 
at  stake  during  the  season  (Lillian  Vernon  takes 
37  percent  of  its  orders  over  the  Web)  putting 
off  the  launch  is  the  safer  course.  “It  doesn’t 
make  sense  to  add  to  our  general  business  risk 
by  launching  a  new  site  during  our  peak,” 
Shapiro  says. 

Other  projects  on  tap  were  infrastructure- 
related:  migrating  from  Lotus  Notes  to 
Microsoft  Exchange  for  e-mail  and  installing 
PeopleSoft  financials.  Because  there  was  so 
much  work,  several  of  the  projects  were  to  be 
driven  by  user  groups  at  headquarters.  Scott 
shuttled  back  and  forth  between  the  Virginia 
Beach  office  and  the  Rye  headquarters,  but 
could  not  be  in  both  places  at  once.  Shapiro 
agreed  to  serve  as  an  on-the-ground  project 
manager  for  the  DAM  installation. 


Road  Map  for 
Change  Management 

1.  It’s  not  all  B.S  .  Even  if  you  have  been  a  change  management  skeptic 
such  as  CIO  Tom  Scott  in  the  past,  believe  it:  People  have  trouble  assimilating 
change,  and  your  project  is  at  risk  if  you  don't  recognize  that.  But  there’s  a  lot 
you  can  do  to  help  the  organization  over  the  hurdles. 

2.  Show  them  the  big  picture.  This  includes  why  the  company  is 
undertaking  the  effort  and  why  it  is  important  for  business  goals.  Also,  talk 
about  the  normal  course  of  projects  and  the  inevitable  ups  and  downs  all  proj¬ 
ects  go  through.  Do  not  assume  anyone  has  been  through  a  major  IT  project 
before. 

3 .  Show  them  the  small  picture.  Let  each  individual  know  how  his 

own  job  will  change  as  a  result  of  the  project.  Do  not  ever  give  the  blanket 
statement  “This  is  going  to  make  all  our  jobs  better.” 

4.  Get  in  front  of  your  audience.  Repeat.  Take  for  granted  that 

you’re  going  to  have  to  repeat  these  messages  several  times  to  the  same  peo¬ 
ple.  Change  throws  the  organization  into  doubt  and  turmoil;  people  don’t  lis¬ 
ten  well  when  they’re  threatened. 

5.  Be  honest.  If  some  people  will  lose  their  jobs  or  if  that  decision  remains 
to  be  made,  work  with  HR  to  be  up  front  with  employees.  What  they  don’t  know 
will  hurt  them  and  your  project. 

6.  Don’t  even  begin  without  solid  project  management 

skills  .  Even  if  the  vendor  brings  in  a  project  manager,  designate  an  in-house 
PM  and  make  everyone’s  roles  clear.  -L.G.P. 


What  Went  Wrong 

All  appeared  well  in  the  days  leading  up  to  the 
May  2004  launch  date  of  the  Lago  DAM  sys¬ 
tem.  The  team  was  moving  quickly,  working 
with  the  buyers  to  improve  business  processes. 
However,  just  before  the  system  went  live,  Lil¬ 
lian  Vernon’s  failure  to  take  change  manage¬ 
ment  into  account  became  painfully  evident. 
The  vendor  set  up  classes  for  end  user  train¬ 
ing,  and  the  buyers  and  creative  services  per¬ 
sonnel  who  would  be  affected  dutifully  signed 
up.  Although  they  signed  in  on  the  attendance 
sheet,  many  employees  did  not  bother  to  stay 
for  class.  “They  would  pop  in,  pop  out;  it  was 
as  flagrant  a  revolt  as  I’ve  ever  seen,”  says 
Scott.  These  employees  had  already  made  up 
their  minds  that  the  system  was  not  going  to 
work,  and  they  didn’t  want  any  part  of  it.  In 
retrospect,  Shapiro  and  Scott  admit  they 


should  have  communicated  earlier  and  more 
often  why  the  project  was  necessary  and  how 
it  would  affect  each  individual  specifically. 

Before  the  classes  began,  “we  should  have 
put  everyone  in  a  room  and  said,  Here  is  how 
you  fit  into  this  new  picture,”  Shapiro  says. 
Instead,  the  project  team  fell  back  on  blanket 
statements  that  everyone’s  job  would  be  “bet¬ 
ter”  once  they  had  ways  to  handle  the  product 
photos  and  descriptions  electronically,  as 
opposed  to  manually.  Once  rollout  began, 
however,  they  were  angry  when  their  jobs  were 
harder  instead.  Since  most  had  not  taken  the 
training  seriously,  they  did  not  know  how  to 
use  the  application.  And  many  were  uncertain 
as  to  how  their  jobs  had  changed.  “People  were 
blaming  the  system  for  everything,”  says  Scott. 

The  DAM  team  also  tried  to  look  the  other 
way  when  the  brand-new  business  processes 
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Change  Management 


began  to  break  down.  For  example,  the  proj¬ 
ect  team  had  elected  to  give  exclusive  control 
over  a  catalog  or  website  entry  to  the  buyers. 
But  the  creative  services  people  needed  the 
ability  to  make  changes,  moving  products  to 
different  pages  when  necessary.  For  example, 
a  buyer  might  add  a  Christmas  candle  center- 
piece  to  the  website.  But  the  creative  services 
person  might  need  to  move  that  item  to 
another  location  for  space  reasons.  Since  the 
creative  person  was  initially  not  allowed  to 
make  changes  in  Lago,  this  led  to  frustration. 

“We  allowed  clever  employees  to  create 
workarounds.  But  that  just  caused  more  prob¬ 
lems  down  the  road,”  says  Shapiro.  Untangling 
the  initial  decisions  is  taking  time:  The  com¬ 
pany  was  still  struggling  with  Lago  at  press 
time.  “I  wish  I  could  say  we  are  done  suffer¬ 
ing,”  says  Shapiro.  Yet  the  company  will  stay 
with  Lago  since  he  and  Scott  believe  the  prob¬ 
lems  were  process-  and  people-oriented,  rather 
than  the  result  of  picking  the  wrong  system. 

Repeat  the  Message 

When  Scott  began  work  on  the  Lillian  Vernon 
technology  transformation,  change  manage¬ 
ment  did  not  figure  in  his  list  of  potential  dis¬ 
asters.  With  so  much  work  ahead,  the  softer 
side  wasn’t  on  his  early  radar  screen.  “Here’s 
what  1  was  thinking:  We  have  third-party 
packages.  We’re  bringing  in  vendors  who  will 
bring  change  management  expertise  to  the 
table.  We  have  capable,  gung-ho  teams.  Gid- 
dyap,  let’s  go,”  says  Scott.  In  other  words, 
launch  the  projects  and  fix  problems  later. 

At  that  time  Scott  was  not  a  big  believer  in 
change  management  as  a  discipline.  In  previ¬ 
ous  jobs,  he  had  seen  enough  Big  Five  firms 
come  in,  tell  you  to  do  a  project  newsletter  and 
hold  brown-bag  lunches  with  employees,  and 
then  charge  five-figure  fees.  It  seemed  like  a 
lot  of  time  and  money  spent  for  not  much 
return.  But  he  rapidly  shifted  his  view  of 
change  management  because  the  problems 
were  not  confined  to  Lago.  The  toll  the  change 
was  taking  on  management  and  the  user  com¬ 
munity  had  become  evident  by  late  summer. 
“They  would  start  raising  the  question  of 
whether  we  should  be  on  this  new  system  or 
go  back,”  says  Scott.  Enterprisewide,  much 
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SYSTEM 

INITIAL  CAPITAL  OUTLAY 

PAYBACK  PERIOD 

Ecometry  order-entry, 
customer  service  and 
warehouse  management  system 

$2.1  million 

2  years 

ILOG  shop  floor  loading 
and  scheduling  software 

$500,000 

12  months 

Merkle  customer  database 

$1.2  million 

12  months 

Evant  merchandise 
planning  system 

$750,000 

12  months 

Comosoft  Lago  digital 
asset  management  system 

$500,000 

18  months  to  2  years 

New  website  (Fry) 

$600,000 

12  months 

PeopleSoft  financials 

$300,000 

12  months 

Infrastructure 

$1.1  million 

2  years 

Total  spend 

$7.05  million 

time  was  spent  rehashing  problems  such  as 
the  DAM  snafu  rather  than  solving  them. 

“What  I  mistook  for  them  being  ready  to 
go  was  that  they  just  didn’t  know.  I  found  out 
partway  through  that  a  lot  of  the  employees 
had  never  been  through  a  single  systems  proj¬ 
ect,”  says  Scott.  It  was  essential  to  communi¬ 
cate  more  clearly  to  employees,  to  speak  to 
them  as  individuals  well  in  advance  of  the 
project  getting  under  way.  Scott  has  learned 
that,  far  from  being  a  frill,  basic  communica¬ 
tion  creates  the  underpinning  for  a  successful 
implementation.  The  essence  of  change  man¬ 
agement,  in  his  view  is  “a  few  well-placed, 
well-delivered  conversations  to  the  right  audi¬ 
ence.  And  then  you  follow  up,  again  and 
again,”  he  says.  “There’s  not  a  whole  lot  more 
to  it.”  Bowing  to  the  change  management 
gurus,  his  project  office  started  publishing  a 
project  review  newsletter  in  July,  which  he 
believes  has  gone  a  long  way  toward  helping 
employees  understand  how  they  and  the  com¬ 
pany  stand  to  benefit  from  the  project.  “You 
have  to  say  why  you  are  doing  this — not  once, 
but  many  times,”  says  Scott. 

The  good  news  is  that  Scott  has  put  the  les¬ 
sons  learned  immediately  to  work  while  over¬ 
seeing  the  other  projects.  Deployment  of  the 
most  important  system — the  application  for 
order  entry,  customer  service  and  warehouse 


management — is  only  one-third  complete. 
But  the  implementation  so  far  has  already 
gone  much  more  smoothly  than  the  DAM 
rollout.  “We  addressed  change  right  from  the 
beginning.  Accountability  was  clear;  not 
knowing  where  they  stand  drives  people 
crazy,”  says  Scott.  He  also  appointed  a  certi¬ 
fied  project  manager  to  each  project. 

Not  surprisingly,  the  time  line  for  the  proj¬ 
ects  has  slipped.  Whereas  Shapiro  and  Scott 
had  once  envisioned  having  everything  com¬ 
plete  by  the  end  of  this  year,  now  only  the 
DAM  system  and  the  website  are  complete. 
The  merchandise  planning  system  installation 
will  be  done  by  March.  And  Scott  says  the 
Ecometry  project  should  be  complete  by 
August. 

But  then  again,  he  knows  now  to  expect  the 
unexpected.  “The  [Ecometry]  project  will 
undoubtedly  go  off  course  several  more  times 
before  we  get  it  in,”  says  Scott.  “You  think  the 
rollout  will  be  like  drawing  a  straight  line 
between  two  points.  But  really  there  are  going 
to  be  lots  of  zigs  and  zags.  You  can  count  on  a 
lot  of  wrong  paths  and  detours.  It’s  just  how 
quickly  we  get  it  back  on  track.”  E3EI 


Lauren  Gibbons  Paul  is  a  freelance  writer  based 
in  Waban,  Mass.  She  can  be  reached  at  lauren.paul 
@comcast.net. 
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The  old  effort  to  figure  out  IT’s  impact 
on  organizational  productivity  has  new 
life.  Leaving  macroeconomic  theories  to 
others,  some  CIOs  are  applying  IT  pro¬ 
ductivity  measures  to  operations  at  their 
own  companies.  At  Cisco  Systems,  for 
instance,  a  cross-departmental  council 
has  been  working  to  hash  out  metrics  for 
improving  business  process  operations, 
according  to  CIO  Brad  Boston. 

In  2003,  the  council  developed  metrics 
to  evaluate  the  efficiency  of  its  online 
order  processing.  Less  than  30  percent  of 
orders  were  getting  automatically  routed 
to  manufacturing,  since  high  error  rates 
necessitated  manual  input,  according  to 
Boston.  “We  mapped  the  process  of 
orders  coming  in  and  developed  metrics. 
We  optimized  the  process,  put  new  tools 
in  and  doubled  the  percentage  of  orders 
that  went  directly  to  manufacturing 
within  six  to  nine  months,”  he  says. 

Cisco  also  relies  heavily  on  IT  pro¬ 
ductivity  metrics  for  budgeting,  particu¬ 
larly  in  negotiating  contract  labor.  “We 
use  [productivity  measures]  for  budget¬ 
ing  for  development  projects  to  decide 
how  much  to  do  with  internal  staff  and 
how  much  with  contract  labor,”  he  says. 
“And  we  do  follow-up  evaluations  of  the 
actual  delivery  at  total  cost.” 

At  a  time  when  CIOs  are  under 
greater  pressure  than  ever  to  prove  the 
value  of  IT,  productivity  metrics  that 
measure  the  “true”  value  of  IT  are  the 
Holy  Grail.  When  it  comes  to  IT,  pro¬ 
ductivity  is  everywhere  and  nowhere. 
The  axiom  still  holds  that  the  effects  of 
IT  show  up  everywhere  except  in  pro¬ 
ductivity  statistics,  to  paraphrase  econo¬ 
mist  Robert  Solow. 

CIOs  are  inching  toward  better  high- 
level  measures  that  relate  to  overall  busi¬ 


ness  productivity,  but  they  are  finding  it 
easier  to  collect  empirical  data  about 
throughput,  uptime,  service  levels  and  the 
like.  A  new  twist  in  productivity  metrics  is 
the  brave  new  world  of  business  process 
design  and  analysis.  CIOs  are  using  (and 
in  some  cases  creating)  performance  met- 


Cisco  Systems  CIO  Brad  Boston  spends  a 
fifth  of  his  time  improving  IT  productivity— 
and  thereby  business  results. 


rics  to  help  their  organizations  get  a  better 
handle  on  partnerships  and  service  agree¬ 
ments,  and  to  claim  (or  keep)  a  place  at 
the  strategic  planning  table. 

CIOs  say  that  these  emerging  “value” 
metrics — so  termed  because  they  gauge 
the  value  that  IT  contributes  to  the  enter¬ 
prise — are  helping  them  to  better  manage 
their  operations,  and  to  communicate  and 
coordinate  with  line-of-business  and  sen¬ 
ior  management. 

Value  metrics  occupy  the  intersec¬ 
tion  of  hard  and  soft  performance 
measures,  combining  observation  and 
intuition.  “Most  clients  I  work  with  are 
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shifting  their  focus 
away  from  internal 
[IT  productivity]  meas¬ 
ures  and  more  onto 
value  measures,”  says 
Craig  Symons,  a  prin¬ 
cipal  analyst  with  Forrester  Research. 
“They’re  asking,  What  value  are  we  con¬ 
tributing  to  revenue  growth,  to  being 
able  to  roll  out  new  products  and  serv¬ 
ices,  to  cut  the  time  to  do  those  things, 
add  to  customer  satisfaction  and  reten¬ 
tion  and  cut  costs?” 


Productivity’s  Common  Currency 

While  acknowledging  the  limitations  of 
productivity  metrics,  CIOs  use  them  to 
strategize,  budget,  weigh  outsourcing  deci¬ 
sions,  run  (and  terminate)  projects  and 
communicate  with  other  senior  executives. 

Unsurprisingly,  technology-sector  CIOs 
tend  to  be  more  advanced  in  their  think¬ 
ing  about  productivity  metrics  than  their 
peers  in  other  industries. 

“Directly  or  indirectly,  probably  10  to 
15  percent  of  my  time  is  tied  up  in  this,” 
says  Intel  Vice  President  and  CIO  Doug 
Busch.  “A  huge  fraction  of  what  we  do  in 
IT  is  aimed  at  improving  productivity, 
either  task  productivity  or  knowledge 
worker  productivity.  [But]  I  think  as  an 
industry  we’re  kind  of  thrashing  about 
on  this  problem.” 

For  Cisco’s  Boston,  the  figure  is  closer 
to  20  percent.  “Some  of  that  is  focused 
on  the  cost  of  contract  labor,  but  it’s 
mostly  making  business  more  produc¬ 
tive,”  he  says. 

CIOs  are  awash  in  performance  data: 
internal  IT  measures  of  hardware,  soft¬ 
ware,  network — not  to  mention  person¬ 
nel,  departmental  and  organizational 
ratings  of  IT  services,  vendor  and  service 
provider  claims  (and  actual  results). 

Traditional  high-level  measures  of 
productivity  include  IT  budget  as  a  per¬ 
cent  of  total  revenue,  typically  compared 
with  peer  organizations  in  the  same 
industry  and  the  ratio  of  IT  staff  to  total 


employees.  Other  staffing  ratios  count 
servers  or  PCs  per  support  person. 

How  Many  Programmers  Can 
Dance  on  the  Head  of  a  Pin? 

It’s  always  been  easy  to  lose  the  forest  for 
the  trees  when  considering  productivity. 
But  especially  now  that  IT  statistics  are 
so  easy  to  come  by,  CIOs  have  to  think 
carefully  about  what  they  want  to  get  out 
of  the  metrics. 

“Personally,  I  couldn’t  care  less  how 
many  lines  of  code,  how  many  tasks  it 
runs,”  says  Sinclair  Stockman,  CIO  of 
British  Telecom.  “If  the  system  is  coming 
out  earlier  than  they  told  me  it  would,  and 
I’m  able  to  deliver  more  benefit  than  I 
thought  I  could,  and  it’s  going  to  cost  me 
less,  I’m  very  happy.” 

According  to  Stockman,  getting  the 
most  out  of  IT  systems  depends  on  “the 
efficiency  and  effectiveness  of  the  process 
change,  the  training  process,  and  the  ruth¬ 
lessness  with  which  you  actually  go  after 
the  business  benefits  that  were  delivered 
as  a  result  of  the  systems  being  deployed.” 

This  casts  productivity  in  terms  of 
business  change.  The  trick  is  relating 
individual  project-level  performance  to 
bottom-line  results.  “Between  small 
teams  and  the  very  top  level  where  you’re 
looking  at  the  business  benefits  of  mas¬ 
sive  programs,  there’s  this  massive  chasm,” 
says  Stockman. 

The  one-to-one  communication  that’s 
required  at  the  top  level  of  an  organiza¬ 
tion  often  injects  less  quantifiable  elements 
into  the  productivity  mix,  according  to 
Andy  Rowsell-Jones,  a  Gartner  vice  pres¬ 
ident  and  research  director. 

IT  productivity  depends  on  the  CIO’s 
personal  standing  and  political  skills, 
he  says.  “You’ve  got  to  have  that  rela¬ 
tionship  piece  and  credibility  piece.  Do 
you  get  asked  about  non-IT  things,  invited 
to  product  team  meetings  as  a  possible 
source  of  ideas?  These  are  soft,  squishy 
metrics.  But  at  that  level,  the  personal  rela¬ 
tionship  is  politically  important.” 


If  a  CIO’s  executive  relationships  are 
weak,  “there’s  a  greater  chance  that  IT 
gets  reduced  to  an  equation  of  cost  of 
service,”  he  says.  “Then  you’ve  got  to 
look  out  because,  for  accountants,  service 
is  never  good  enough  and  cost  is  always 
too  high.” 


MANAGEMENT  RESEARCH 

Takingthe 
Measure  of  IT 
Performance 

At  the  more  abstract  and  macroeco¬ 
nomic  end  of  IT  productivity  metrics, 
the  Information  Work  Productivity 
Council  (IWPC)  is  hashing  out  method¬ 
ologies  of  its  own  for  internal  use  and 
consulting  purposes.  IWPC-backed 
research  is  delving  into  the  link  between 
business  performance  and  IT  invest¬ 
ments;  individual  employee  skill  sets 
and  roles;  and  how  information  (bits 
and  social  interactions)  moves 
through  an  organization. 

The  group— which  includes 
Microsoft,  Cisco,  Accenture,  HP,  Xerox 
and  Intel— is  working  on  productivity 
benchmarks  for  finance,  retail  and 
manufacturing  industries,  a  methodol¬ 
ogy  for  tracking  information  use  and 
benefits,  and  equations  for  sizing  up 
organizational  capacity.  But  it’s  likely 
to  be  a  while  before  such  metrics  find 
their  way  to  CIOs’  desks,  say  analysts. 

-T.S.B. 


Pitfalls  of  Productivity 

Ironically,  the  preoccupation  with  pro¬ 
ductivity  can  be  a  drain  on  productivity. 

“We  don’t  have  a  good  way  to  [calcu¬ 
late  IT  productivity],  either  to  forecast  or 
to  measure,”  Intel’s  Busch  says.  “As  a 
result,  we  end  up  justifying  and  rejustify¬ 
ing,  and  analyzing  and  reanalyzing  stuff, 
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and  coming  at  it  from 
different  angles  and 
trying  to  persuade 
people  with  different 
measures.” 

When  it  comes  to 
rating  knowledge  workers,  simple  for¬ 
mulas  tend  to  miss  the  point.  “I  don’t 
think  personnel  appraisals  are  as  depend¬ 
ent  on  productivity  metrics,”  Busch  says. 
“We  measure  people  on  results  rather 
than  on  unit  output  per  period  of  time. 
At  one  level  you  can  say,  Yes,  productiv¬ 
ity  has  a  huge  impact  on  that.  But  it’s 
kind  of  an  indirect  impact. 

“Making  a  smart  decision  that  pro¬ 
duces  the  best  result  in  the  shortest  period 
of  time  is  really  what  we’re  looking  for 
rather  than  how  fast  you  turn  a  crank,” 
he  says. 

One  of  the  problems  in  quantifying  the 
role  of  IT  in  business  is  the  one-off  nature 
of  so  many  projects  and  processes.  “It 
may  sound  a  little  philosophical,  but  I’m 
not  sure  whether  you  can  talk  about 
productivity  in  a  world  where  you’re  not 
actually  talking  about  repeatable  tasks,” 
BT’s  Stockman  says. 

While  it  may  be  relatively  straight¬ 
forward  to  gauge  the  productivity  of 
offshore  software  houses  that  bang  out 
low-level  code,  it’s  tougher  to  quantify  the 
business  benefit  delivered  by  an  in-house 
IT  team.  If  metrics  are  unable  to  capture 
that  value,  it  can  prejudice  managers 
toward  outsourcing. 

Measuring  productivity  of  in-house  IT 
boils  down  to  tying  it  to  business  change. 
“You  need  to  ensure  that  there  are  pro¬ 
grams  of  activity  in  place  which  actually 
effect  business  benefit  improvements  for 
the  company,  and  there  are  some  meas¬ 
ures  you  can  apply  to  that,”  Stockman 
says.  “How  much  business  benefit  have 
you  delivered?  How  many  people  did  it 
take  you  to  do  that?  What  was  the  level 
of  investment  that  you  put  in?  What 
caused  it  to  be  good?  What  caused  it  to 
be  bad?” 


MANAGEMENT  REPORTS 

The  End  of  Service  with  a  Smile? 


Outsourcing  trends 
erode  the  commitment  of 
top  service  employees 

No  matter  how  good  your  IT  systems, 
the  professionalism  and  knowledge  of 
your  customer-facing  staffers  have  a 
major  impact  on  the  reputation  of  IT. 
Surly  and  unhelpful  support  staff  have 
given  many  a  company  a  black  eye. 

A  recent  study  by  the  Service  and 
Support  Professionals  Association 
(SSPA)  looks  at  how  companies  can 
attract  and  keep  "top  talent” 
support  staff— defined  as 
the  top  10  percent  of 
customer-facing 
employees  at 
technology  com¬ 
panies.  The 
study’s  lessons 
apply  not  only 
to  tech  support 
staffers  dealing  with 
mass-market  con¬ 
sumers  but  also  extend  to 
IT  staff  supporting  internal  users. 

It  turns  out  that  money  is  not  the 
reason  the  top  talent  joins  or  stays 
committed  to  an  employer.  They  tend 
to  be  attracted  to  jobs  by  challenging 
work  rather  than  pay  levels,  and  they 
tend  to  be  motivated  by  recognition 
rather  than  money,  according  to  the 
study  of  500  top  employees  at  300 
SSPA  member  companies.  (Member¬ 
ship  inclines  toward  technology  and 
consulting  companies  such  as  Cisco, 
Deloitte  Touche  Tohmatsu,  IBM  and 
Oracle.)  To  attract  and  retain  top 
talent,  the  SSPA  recommends  two 
strategies  in  particular:  extensive 
training  opportunities  and  formal 
development  programs  that  offer  new 
challenges  and  a  path  up  the  ladder. 


These  strategies  seem  likely  to 
become  less  prevalent,  however,  as 
outsourcing  and  offshoring  of  sup¬ 
port  jobs  accelerates.  CIOs  these 
days  seek  to  lower  their  support 
costs  rather  than  spend  money  on 
training  and  development.  Indeed, 
in  a  recent  study  by  SSPA  and 
Tech  Strategy  Partners  that  queried 
200-plus  IT  executives  in  enterprises 
of  various  sizes,  58  percent  were 
willing  to  send  tech  support  offshore. 

Their  primary  concerns  with 
tech  support,  not  sur¬ 
prisingly,  were  value 
and  cost  reduction. 
But  these  CIOs 
are  at  odds  with 
end  consumers. 
The  same  SSPA 
study  asked 
more  than  2,000 
mass-market  end 
consumers  (that  is, 
people  in  their  homes) 
for  their  views  on  tech  support. 
Nearly  half  said  that  they  had  experi¬ 
enced  poor  customer  service  skills 
from  offshore  representatives, 
compared  with  only  12  percent  from 
onshore  tech  support  employees. 

End  consumers  over  45  years  old 
were  overwhelmingly  opposed  to 
offshore  support,  even  if  service 
levels  were  high. 

CIOs  rarely  work  directly  with  end 
consumers:  those  are  the  customers 
of  CIOs’  customers.  Yet  the  SSPA 
findings  raise  a  vexing  question:  Is 
offshore  outsourcing  a  short-term 
solution  to  cost  pressures  that  is 
creating  a  longer-term  problem  with 
customer  satisfaction  and  loyalty? 

-Edward  Prewitt 
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Nonetheless,  the  more  closely  the  IT 
organization  participates  in  business 
process  design,  the  harder  it  becomes  to 
distinguish  its  contributions.  “One  of  the 
most  fundamental  complications  in  this 
whole  productivity  question  is:  What’s  the 
impact  of  business  process  and  what  is  the 
impact  of  the  tools  that  enable  the  busi¬ 
ness  process?”  Busch  says.  “It’s  very  diffi¬ 
cult  to  disentangle  the  two,  and  I  frankly 
have  come  to  the  conclusion  that  you 
shouldn’t  even  try  to.  You  ought  to  look  at 
them  as  a  whole.” 

Looking  ahead,  CIOs  say  they  antici¬ 
pate — sometime  in  the  next  five  years  or 
so — to  see  some  standardized  perform¬ 
ance  metrics  that  more  closely  tie  IT  to 
business  value. 

Productivity  metrics  are  morphing  to 
fit  the  environment.  The  key  is  finding  a 
way  to  establish  a  productivity  baseline, 
according  to  Busch.  “I  don’t  think  we’re 
very  close  to  it,”  he  says.  “It’s  hard  to  tie 
[economic  measurement  of  companies’ 
results]  back  to  individual  project  activity 
in  a  lot  of  cases.  We  need  something  that’s 
much  more  trackable.”  EE3 


Ted  Smalley  Bowen  ( ted_bowen@hotmail.com ) 
is  a  Boston-based  freelancer  covering  technology. 


Leadership  Agenda  by susan  h.cramm 

Be  Better  Than 
You  Really  Are 

How  to  understand  your  personality  type  for  leadership  success 

Sara  is  a  divisional  CIO  who  has  decided  she’s  not  cut  out 
for  the  top  job.  She  doesn’t  think  she  has  the  full  comple¬ 
ment  of  skills  to  be  a  good  CIO.  It’s  a  shame  because 
Sara  is  a  seasoned  IT  professional  who  is  strategic, 
decisive,  results-oriented  and  skilled  at  directing  the 
activities  of  others.  While  it’s  true  that  she  doesn’t  have 
all  of  the  broad  and  diverse  skills  required  for  the  CIO 
job,  it’s  also  true  that  most  people  would  fail  to  make  the  cut.  Like  every 
leadership  job,  the  CIO  role  is  big— too  big  for  one  person.  The  good  news 
is  that  you  can  improve  yourself  to  fill  the  space  required  by  the  CIO  job. 

CIOs,  like  other  executives,  often  fail  in  their  responsibilities  because 
they  don’t  have  the  self-awareness  and  humility  to  mitigate  their  weak¬ 
nesses  and  avoid  overusing  their  strengths.  Very  few  of  us,  in  fact,  are  able 
to  see  ourselves  through  the  eyes  of  others.  We  require  some  kind  of  exter¬ 
nal  feedback. 

A  useful  tool  in  this  regard  is  the  familiar  Myers-Briggs  personality  type 
indicator  (known  as  MBTI).  Isabel  Briggs  Myers  explained  that  “each  of  us  has  a 
set  of  gifts,  a  set  of  mental  tools  that  we  have  become  comfortable  using  and 
thus  reach  for  in  the  everyday  business  of  living”  (from  her  book,  Gifts  Differing: 
Understanding  Personality  Type).  Most  people  know  their  MBTI,  but  few  can 
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articulate  how  their 
type  should  organ¬ 
ize  resources  and 
approach  their  jobs. 

Sara  has  a  person¬ 
ality  type  common  to 
many  IT  professionals  and  executives: 
She  is  an  INTJ  (introverted,  intuitive, 
thinking,  judging).  INTJs  typically  are 
imaginative,  determined  innovators 
who  are  stimulated  by  difficulties  and 
attracted  to  bigger  and  bigger  chal¬ 
lenges.  INTJ  is  also  the  most  independ¬ 
ent  of  all  personality  types. 

These  tendencies  have  profound 
implications  for  what  Sara  needs  to 
do  to  succeed,  both  as  a  divisional  CIO 
and  as  an  aspirant  for  the  top  job.  By 
examining  the  CIO  success  criteria 
and  understanding  the  implications 
of  Sara's  type,  we  can  learn  how  she 
can  become  better  than  she  really  is. 

INTJs  and  the  Five  Criteria 
for  CIO  Success 

1  Defining  and  communicating  a 
shared  IT  vision,  strategy  and 
tactical  objectives.  Sara’s  intuitive 
orientation  means  that  she  has  the 
insight  to  identify  issues  and  patterns, 


and  the  inspiration  and  imagination  to 
define  a  better  approach  to  IT.  Unfortu¬ 
nately,  her  introverted  nature  means 
that  she  treats  strategy  making  as  a 
solitary  pursuit.  Setting  direction  is  a 
participative  process;  otherwise,  the 
commitment  necessary  for  action 
will  be  missing.  To  guard  against  her 
natural  tendencies,  Sara  must  define  a 
process  that  ensures  the  participation 
of  key  stakeholders  across  the  organi¬ 
zation.  She  must  then  assign  IT  profes¬ 
sionals  who  are  intuitive  extroverts 
(ENs,  in  the  Myers-Briggs  shorthand) 
as  key  resources  in  the  effort. 

CIOs,  like  other 
executives,  often 
fail  in  their 
responsibilities 
because  they 
don’t  have  the 
self-awareness  and 
humility  to  mitigate 
their  weaknesses 
and  avoid  overusing 
their  strengths. 


2  Delivering  quality  results  on  time 
and  within  budget.  Sara's  judgmen¬ 
tal  nature  means  she  is  comfortable  with 
planning  and  making  decisions  and  is 
determined  to  get  things  done.  She  is 
stimulated  by  difficulties  and  loves  to 
solve  problems.  But  since  INTJs  get 
bored  with  details  and,  at  times,  do  not 
live  in  the  world  of  facts,  Sara  needs  to 
refine  her  thinking  process.  She  must 
ensure  that  her  project  and  operational 
leaders  are  not  people  just  like  herself,  or 
there’s  a  good  chance  that  her  IT  organi¬ 
zation  won’t  deliver  on  time  or  meet  the 
needs  of  the  business.  Unlike-minded 
people  will  act  as  “sensors”  to  keep  the 
project  grounded  in  reality. 

Helping  the  business  realize  value 
from  IT  investments.  IT  value  is 
a  dual  challenge:  First,  governance 
needs  to  be  established;  second,  busi¬ 
ness  leaders  need  to  be  convinced  that 
following  the  rules  is  in  their  best  inter¬ 
est.  Sara  has  no  problem  defining  what 
mechanisms  are  necessary  to  promote 
value  realization,  but  she  does  not 
inherently  possess  the  gifts  of  persua¬ 
sion  to  get  the  rest  of  the  organization 
on  board.  To  balance  her  strengths  and 
shore  up  her  weaknesses,  she  will  need 
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well  at  night. 


How  do  you  make  short-term  cuts  without  losing  the  long¬ 
term  view?  What  are  the  rules  of  smart  IT  spending?  How 
do  you  fund  innovations  during  hard  times?  Turn  to  the  CIO 

FOCUS™  on  I  T.  COST  CONTROL:  SMARTER  SPENDING 
STRATEGIES  FOR  TIGHT  TIMES— actionable  information  cre¬ 


CIO  FOCUS™ 

IT  Value:  Measurement  Tools 
and  Techniques  That  Work 


ated,  filtered  and  packaged  by  the  award-winning  editors  of 
CIO  magazine. 


CIO  FOCUS™  is  delivered  right  to  your  desktop,  giving  you 
immediate  access  to  the  information  you  need.  And  for  your 
future  reference  needs,  the  electronic  file  is  followed  by  a 
packaged  version,  shipped  within  72  hours. 


Software  Vendor  Relationships: 
Selecting,  Vetting  and 
Managing  Partners 

Fundamentals  of  the  CIO  Role 

Applied  Wireless:  Making 
Wireless  Work  in  Business 


CIO  FOCUS" 

STRATEGIC  GUIDES  FOR  EXECUTIVE  DECISION  MAKING 


The  Resource 
for  Information 
Executives 


FOR  EXECUTIVE  DECISION-SUPPORT  TOOLS,  VISIT  THE  CIO  STORE-THE  CIO'S  KNOWLEDGE  MARKETPLACE. 

www.TheCIOStore.com 


Hot 

Seat 


extroverted  thinkers 
(ETs)  on  her  staff 
who  feel  comfortable 
with  the  collaborative 
process  of  negotiation 
but  who  will  not  give 
in  on  key  principles  or  avoid  conflict  just 
to  make  everybody  happy. 

4  Fostering  good  relationships. 

Since  Sara  is  more  comfortable 
with  ideas  than  with  people,  she  tends  to 
make  decisions  without  considering  the 
feelings  of  others.  She  often  approaches 
situations  with  her  mind  made  up  and  is 
surprised  by  the  opposition  that  results. 
To  be  sure,  quick-and-easy  relationships 
will  never  be  Sara’s  strong  suit,  but  she 
needs  to  at  least  neutralize  her  natural 
tendencies  by  forcing  herself  to  interact 
with  others— even  though,  in  her  mind, 
there  is  no  real  purpose  to  it.  She  can 
force  herself  into  business  relationships 
by  making  it  a  project  (for  example, 
defining  whom  she  needs  to  talk  with 
and  scheduling  regular  interactions), 
improving  her  cocktail  party  conversa¬ 
tional  skills  (using  reflective  and  active 
listening)  and  refining  her  negotiation 
skills  (using  principled  rather  than 
positional  negotiating). 


Once  again,  Sara  can  use  her  staff 
to  bolster  her  weaknesses.  In  this  case, 
she  can  ensure  that  her  client-facing 
personnel  includes  an  adequate  num¬ 
ber  of  extroverts. 

5  Building  and  leading  a  credible 
IT  organization.  Executives  also 
derail  when  they  fail  to  staff  correctly. 
Since  introverted  thinkers  (ITs)  such  as 
Sara  connect  to  others  through  ideas 
rather  than  feelings,  they  don’t  get  to 
know  people  on  a  human  level.  There¬ 
fore,  they  have  a  tendency  to  appeal  to 
the  head  and  hands,  but  not  the  heart. 
Sara’s  people  are  stimulated  by  her 
vision,  clear  direction  and  delegated 
authority,  but  they  feel  their  personal 
needs  are  not  necessarily  understood. 
Their  assignments  may  not  be  relevant 
to  their  career  aspirations.  For  Sara, 
the  answer  is  to  carve  out  time  to  get 
to  know  others  by  asking  questions 
about  their  goals,  values,  and  perceived 
strengths  and  weaknesses.  Then  she 
can  apply  her  intuitive,  thinking  skills  to 
define  a  development  plan  that  will 
meet  the  needs  of  both  individuals  and 
the  organization. 

Each  of  you  shares  Sara's  predica¬ 
ment,  if  with  a  different  mix  of  personal¬ 


ity  tendencies.  Every  day,  your  talents 
and  weaknesses  fight  each  other  for 
their  chance  in  the  spotlight.  How  well 
you  fulfill  your  role  with  the  necessary 
skills— whether  your  skills  or  others’— 
defines  how  successful  you  are  and  how 
much  you  enjoy  what  you  do. 

Dust  off  your  latest  personality 
assessments  and  extract  the  insights 
necessary  to  become  better  than  you 
really  are.  The  MBTI  isn’t  the  only 
assessment,  nor  are  such  tools  the 
only  method  of  learning  your  strengths 
and  weaknesses.  Make  sure  you  work 
for  a  variety  of  bosses  throughout  your 
career.  Sign  up  for  the  right  kind  of 
360-degree  assessment,  so  that  you 
are  able  to  validate  your  progress. 

Armed  with  self-knowledge  and  with 
their  egos  in  check,  executives  such  as 
Sara  (and  you)  can  use  every  resource 
and  tool  to  become  better  than  they 
really  are.  BE! 


For  reader  questions  and  answers  from  Susan  H. 
Cramm,  go  online  to  www.cio.com/leadership/ 
agenda.html.  Cramm  ( susan@valuedance.com ) 
is  the  founder  and  president  of  Valuedance,  an 
executive  coaching  firm  in  San  Clemente,  Calif., 
and  a  former  CIO  and  CFO. 
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When  Bad  Things  Happen 

to  Good  Projects 

By  Christopher  Koch  I  50 

Hewlett-Packard  had  contingency 
plans  in  place  when  it  migrated 
one  of  its  biggest  North  American 
divisions  onto  a  centralized  ERP  system.  It 
planned  for  three  weeks’  worth  of  IT  snafus 
and  laid  in  three  weeks’  worth  of  extra 
server  products.  But  the  plan  wasn’t  nearly 
robust  enough.  When  the  system  went  live, 
as  many  as  20  percent  of  customer  orders 
stopped  dead  in  their  tracks  between  the 
legacy  order-entry  system  and  the  new  ERP 
system.  And  HP  lacked  sufficient  manual 
workarounds  to  keep  products  flowing 
fast  enough  to  meet  demand.  In  August, 
the  CEO  pegged  a  $  1 60  million  loss  to  the 
impact  of  the  order  backlog.  This  disaster, 
like  many  that  result  from  the  vastly  mcreased 
risks  that  major  enterprise  software  projects 
pose  to  businesses  with  high-volume  supply 
chains,  could  have  been  prevented — not  by 
trying  to  eliminate  every  possibility  for  proj¬ 
ect  error,  but  by  taking  a  broader,  worst- 
case  view  of  the  business  impact  of  IT 
failure.  CIOs  must  convince  fellow  execu¬ 
tives  of  the  risk  and  drive  business  contin¬ 
gency  plans  that  are  as  robust  as  the  IT 
project  plans  themselves. 


“We  had  a  series  of 
small  problems,  none 
of  which  individually 
would  have  been  too 
much  to  handle.  But 
together  they  created 
the  perfect  storm.” 

-HEWLETT-PACKARD  CIO 
GILLES  BOUCHARD 


Reforming  California  IT  By  Susannah  Patton  I  60 

J.  CLARK  KELSO— a  CIO  with  no  formal  IT  background,  running  IT  without  a  central  depart¬ 
ment  and  little  executive  authority — is  trying  to  reform  and  rebuild  California’s  IT,  which  was  devas¬ 
tated  by  the  2001  Oracle  no-bid  contract  scandal  that  cost  the  last  state  CIO  his  job.  Kelso,  a  legal 
scholar  and  part-time  law  professor,  has  used  his  ability  to  read  people  and  situations  to  defuse  the  cri¬ 
sis  and  bring  the  state’s  feuding  TT  leaders  together.  He’s  made  progress  on  consolidation  projects  that 
have  languished  for  years,  and  he’s  forged  ties  to  a  skeptical  state  legislature.  With  Governor  Arnold 
Schwarzenegger’s  backing,  Kelso  has  been  able  to  move  forward  with  plans  to  restart  e-government 
initiatives  and  wring  efficiencies  from  IT  procurement  practices  that,  until  now,  have  been  wastefully 
redundant.  He  has  also  drafted  a  five-year  strategic  plan  for  California  IT  to  replace  the  state’s  aging 
legacy  systems  with  integrated  ERP-type  systems.  Kelso’s  success  in  getting  people  to  work  together 
in  a  fiercely  divided  political  climate  provides  an  instructive  tale. 


How  to  Be  a  Mind  Reader  By  Meridith  Levinson  I  72 

ACCURATELY  INTERPRETING  THE  MEANINGS  of  nonverbal  communications,  especially 
facial  expressions,  can  make  CIOs  more  effective  leaders  and  managers,  says  Paul  Ekman,  psycholo¬ 
gist  and  author  of  Emotions  Revealed:  Recognizing  Faces  and  Feelings  to  Improve  Communication 
and  Emotional  Fife.  Reading  facial  expressions  is  a  particularly  useful  skill  for  business  executives 
because  in  business  settings  people  often  don’t  say  what  they  really  think.  If  CIOs  could  recognize 
how  different  emotions  manifest  themselves  on  the  face,  they’d  be  able  to  discern  much  more  quickly, 
for  example,  when  an  individual  is  starting  to  get  angry.  This  knowledge  and  ability  can  make  CIOs 
more  aware  of  unspoken  political  tensions  during  board  or  executive  committee  meetings.  We  show 
you  how  to  get  started  and  offer  an  online  quiz  ( www.cio.com/printlinks )  to  help  you  test  your  abilities. 

Time  to  Change  By  Lauren  Gibbons  Paul  I  78 

FOLLOWING  THREE  YEARS  OF  DECLINING  REVENUE  and  an  $18  million  loss  in  2003, 
the  new  owners  of  catalog  retailer  Lillian  Vernon  aimed  to  beef  up  sales  by  freshening  the  product 
line,  adding  a  fleet  of  sales  representatives  and  improving  the  shopping  experience  in  every  channel. 
That  required  a  complete  overhaul  of  IT,  and  the  new  management  team  decided,  fatefully,  that  speed 
was  crucial  to  the  turnaround;  10  major  applications  had  to  be  replaced  by  the  end  of  2004.  But  it 
didn’t  happen  that  way,  mainly  because  Lillian  Vernon’s  leaders  didn’t  pay  enough  attention  to  change 
management.  In  fact,  CIO  Tom  Scott  initially  disdained  the  “soft  and  squishy”  change  management 
practices  he’d  watched  the  Big  Five  accounting  firms  peddle  in  his  previous  jobs.  Ultimately,  he  real¬ 
ized  his  error  when  users  spurned  training  sessions  and  then  revolted  when  the  new  systems  seemed 
to  make  their  jobs  harder.  Scott  has  put  his  hard-learned  lessons  to  use  on  the  remainder  of  the  rollouts, 
but  the  missteps  caused  the  company  to  miss  its  deadline. 


Hot  Seat:  A  New  Metrics  System  for  IT  By  Ted  Smalley  Bowen  I  89 

AT  A  TIME  WHEN  CIOS  are  under  greater  pressure  than  ever  to  prove  the  value  of  IT,  the  old 
effort  to  figure  out  IT’s  impact  on  organizational  productivity  has  new  life.  CIOs  at  British  Telecom, 
Cisco  Systems  and  Intel  are  using  IT  productivity  measures  to  gauge  the  effectiveness  of  operations 
at  their  own  companies.  A  new  twist  in  productivity  metrics  is  in  business  process  design  and  analy¬ 
sis.  CIOs  say  these  emerging  “value”  metrics — so  termed  because  they  assess  the  value  that  IT  con¬ 
tributes  to  the  enterprise — are  helping  them  to  better  manage  their  operations,  and  to  communicate 
and  coordinate  with  line-of-business  and  senior  management. 
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Best-in-class  IT  organizations  know  that  it  takes  more  than  tools  to  achieve  audit 
compliance,  high  levels  of  service  availability,  and  security.  It  takes  processes  and 
policy  enforcement  as  well.  Organizations  that  implement  best  practices— such  as 
those  from  the  IT  Infrastructure  Library  (ITIL) — in  order  to  build  effective  and 
auditable  change  management  processes,  experience  significant,  quantifiable 
improvements  in  audit  readiness,  service  availability,  security,  and  cost  savings.  The 
rewards  are  worth  it. 

How  do  you  get  there?  Through  an  approach  called  Visible  Ops.  The  Visible  Ops 
Handbook:  Starting  ITIL  in  4  Practical  Steps  offers  a  straightforward  framework  for 
implementing  fundamental  ITIL  processes  around  change  management— the  four 
steps  that,  when  followed,  will  profoundly  improve  IT  operations. 

Tripwire  knows  that  addressing  process  issues  is  never  as  easy  as  addressing 
technology  issues.  So  if  you're  ready  to  take  the  first  step  toward  meeting  your  IT 
performance  goals,  Tripwire  can  offer  you  a  wealth  of  assistance. 

Tripwire  products  and  expertise  have  already  enabled  many  Fortune  1000  companies 
to  significantly  improve  change  management  processes,  and  therefore,  audit 
readiness,  service  availability,  and  IT  cost  reduction.  Tripwire  and  our  authorized 
consulting  partners  can  work  with  you  to  assess  your  current  change  management 
processes  and  plot  the  next  steps  to  take  for  achieving  your  goals. 

I  invite  you  to  read  on  and  enjoy  this  complimentary  book,  courtesy  of  Tripwire  in 
partnership  with  the  ITPI. 


Best  regards 


Gene  Kim 

Visible  Ops  is  a  methodology  that  comprehensively  responds  to  major  issues 
I  have  raised  over  and  over  again  in  my  long  career  in  financial  and 
technology  auditing.  To  attest  to  the  reliability  of  systems,  auditors  need  to 
see:  controls  in  place,  controls  documented,  controls  communicated,  and 
evidence  of  the  controls  in  action.  Visible  Ops  shows  IT  managers  how  to 
build  their  operational  processes  so  they  can  answer  the  auditors'  eternal 
question:  'How  do  we  really  know?' — RUBY  CHRISTINA  BAUSKE,  LEAD 
TECHNOLOGY  AUDITOR,  CPA,  CIA,  CISA,  CISSP 
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Testimonials: 


"A  frequent  complaint  of  ITIL  consultants  is  that  not  many  1TIL  implementation  tools  are  publicly  available.  For 
the  experienced  IT  Service  Management  practitioner,  it  sometimes  seems  we  have  to  re-invent  the  wheel  each 
time.  Visible  Ops  fills  a  big  part  of  that  void.  It  provides  a  practical  insight  in  how  to  kick-start  an  IT  Service 
Management  improvement  effort.  Its  common  sense  approach  and  very  readable  style  give  this  book  a  mandatory 
place  in  the  library  of  any  IT  manager.  Visible  Ops  is  indeed  comparable  to  the  manual  for  an  Emergency  Room  of 
a  hospital.  I  particularly  liked  the  fact  it  does  not  pretend  to  be  an  operational  'bible'  for  the  phases  beyond  the  ER. 

Visible  Ops  describes  four  steps  to  control  an  IT  environment.  The  unassailable  logic  behind  these  steps  is  based 
on  the  practical  experience  of  the  authors,  Gene  Kim  and  Kevin  Behr.  These  same  steps  can  easily  be  mapped  to 
any  maturity  model  and  Visible  Ops  hence  describes  a  roadmap  to  maturity. 

The  first  two  phases  of  Visible  Ops  help  organizations  control  the  infrastructure.  The  third  Visible  Ops  phase  helps 
organizations  control  the  services,  in  the  spirit  of  Service-Oriented  Architectures  and  IT  Service  Management.  The 
last  phase  of  Visible  Ops  helps  organizations  control  the  strategic  value,  which  provides  an  opportunity  for  IT  to 
align  itself  with  the  business  and  to  gradually  maximize  its  'bang  for  the  buck.'  The  easy  mapping  between  the 
Visible  Ops  phases  and  any  maturity  model  validates  the  compelling  logic  of  the  book." — JAN  VROMANT, 
ITSM  CONSULTANT 

"Gene  and  Kevin  have  hit  the  proverbial  'IT  nail'  right  on  the  head.  When  I  educate  customers  on  the  benefits  of 
documented  and  repeatable  procedures  such  as  ITIL  and  COBIT,  they  are  always  concerned  about  the  complexity 
and  where  to  start.  Visible  Ops  creates  a  logical  starting  point  and  details  the  key  'issues  and  indicators.'  This 
handbook  is  a  'must  read'  for  IT  Managers  and  Directors  who  are  implementing  a  mandate  from  their  CIO  or 
Board  of  Directors  to  become  compliant  for  auditors  and  federal  regulations." — HENRY  E.  WOJCIK,  DIRECTOR, 
ENTERPRISE  SERVICE  MANAGEMENT,  NETWORK  DATA  SYSTEMS,  INC 

"The  Visible  Ops  Handbook  is  the  Rosetta  Stone  that  the  IT  industry  and  its  leadership  have  been  seeking  to  allow 
them  to  communicate  the  value  of  ITIL  to  the  business.  Visible  Ops  is  simple  and  clear,  provides  a  roadmap  of 
how  to  make  an  IT  department  not  only  perform  better,  but  also  to  deliver  more  value  back  to  the  business. 
Without  doubt,  each  of  the  four  steps  they  outlined  have  value  and  are  well  supported." — DANIEL  S.  WAITE, 
SENIOR  CONSULTANT,  BMC  SOFTWARE 

"The  Visible  Ops  handbook  provides  a  great  roadmap  for  IT  executives  to  see  their  way  through  the  thicket  of 
chaotic  operations  and  into  the  clearing  of  repeatable  processes.  It  follows  in  the  footsteps  of  software 
development  processes  like  the  Capability  Maturity  Model  (CMM),  and  offers  the  potential  to  provide  a  real  ROI 
by  reducing  the  effort  in  wasted  firefighting."— JEREMY  EPSTEIN,  SENIOR  DIRECTOR,  PRODUCT  SECURITY, 
WEBMETHODS,  INC. 

"Finally,  a  'best  practice'  that  is  based  upon  research  and  industry  knowledge.  Too  often,  best  practice  papers  are 
written  with  little  or  no  influence  or  research  from  those  actually  performing  the  work  in  the  real  world.  This 
approach  is  a  step-by-step,  methodical  approach  for  any  organization  looking  to  get  a  grip  on  Change 
Management  and  improve  operations.  The  books  format  and  introduction  of  methodologies  will  set  the  pace  for 
all  future  publications!"— CHARLES  HORNAT,  GLOBAL  INFORMATION  SECURITY  MANAGER 
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"Visible  Ops  provides  the  IT  practitioner  at  any  level  with  a  catalytic  approach  to  improving  operational  controls. 
The  Visible  Ops  toolset  helps  organizations  find  a  toehold  in  spite  of  sheer  cliffs  of  chaos.  If  you  are  looking  to 
start  or  improve  configuration  management,  champion  a  repeatable  server  provisioning  process,  and  institute 
meaningful  metrics  that  breed  quality  decisions,  Visible  Ops  is  the  place  to  start.  I  recommend  this  to  any  IS 
Management,  as  well  as  any  senior  management  with  a  technical  background  or  IT  staffers  with  management 
ambitions."— BILL  SHINN,  SECURITY  ENGINEER 

"If  you  are  in  IT,  you  are  most  likely  currently  dealing  with  issues  of  firefighting  due  to  rampant  changes,  or  are 
deliberately  ignoring  them  due  to  lack  of  time  and  resources.  Visible  Ops  provides  a  clear-cut  methodology  and 
steps  to  effectively  deal  with  these  issues.  This  book  provides  a  scaleable  template  that  fits  around  any  size  shop 
to  get  back  in  control,  and  then  actually  stay  there.  They  show  you  how  to  regain  control  of  critical  changes, 
whether  it's  an  entire  data  center  rebuild,  a  single  device  failure,  or  upgrading  an  entire  server  farm  to  a  new 
software  release,  and  then  continue  to  manage  them  effectively  from  deployment  to  production  to  retirement." 
—TROY  THOMPSON,  ITIL  CERTIFIED  CONSULTANT 

"This  is  a  very  valuable  resource  for  anyone  just  getting  started.  If  this  resource  had  been  available  when  I  was 
putting  together  the  Change  Management  plan  for  our  department,  it  would  have  saved  me  many  hours  of 
research.  I  highly  recommend  it  as  both  a  reference  and  developmental  tool.  It  will  help  you  identify  the  processes 
and  order  in  which  you  should  develop  and  implement  the  various  ITIL  BS  15000  process  areas.  More  importantly 
the  tips  for  audit  preparation  will  help  you  identify  the  specific  areas  of  improvement  and  help  you  identify  and 
target  areas  requiring  an  organizational  culture  change.  Well  written,  easy  to  follow,  with  good  examples;  It  has 
everything  you  need  from  beginning  development  through  the  measuring  the  results."— JACKIE  SHAFFER, 
SYSTEMS  PROJECT  ADMINISTRATOR,  FLORIDA  DEPARTMENT  OF  EDUCATION 

"In  general,  this  book  provides  a  synopsis  of  the  techniques  and  methodologies  we  at  SIAC  use  to  provide  close  to 
'five  nine'  uptime  for  our  owners  and  customers."— MIKE  PROSPECT,  VICE  PRESIDENT,  SECURITIES  INDUSTRY 
AUTOMATION  CORPORATION 


"Visible  Ops  is  a  methodology  that  comprehensively  responds  to  major  issues  I  have  raised  over  and  over  again  in 
my  long  career  in  financial  and  technology  auditing.  To  attest  to  the  reliability  of  systems,  auditors  need  to  see: 
controls  in  place,  controls  documented,  controls  communicated,  and  evidence  of  the  controls  in  action.  Visible 
Ops  shows  IT  managers  how  to  build  their  operational  processes  so  they  can  answer  the  auditors'  eternal  question: 
‘How  do  we  really  know?'"— RUBY  CHRISTINA  BAUSKE,  LEAD  TECHNOLOGY  AUDITOR,  CPA,  CIA,  CISA,  CISSP 

"Change  management  done  wrong  is  painful,  cumbersome  and  results  in  needless  firefighting.  However,  effective 
change  management  done  correctly  enables  IT  operations  and  information  security  to  work  more  efficiently  and 
better  support  the  business  objectives.  Furthermore,  it  makes  audits  easier  to  pass  and  perform.  The  Visible  Ops 
book  clearly  explains  in  a  practical  and  manageable  approach  what  it  takes  for  organizations  to  implement  change 
management  that  really  works.  If  organizations  agree  to  follow  the  approach  in  this  book  and  stick  to  it,  they  will 
see  how  structured  and  disciplined  change  management  will  actually  make  their  lives  easier,  will  not  stifle 
responsiveness  or  flexibility,  and  will  help  to  extinguish  many  of  the  fires." — CRAIG  MORGAN,  CISSP,  PRINCIPAL 
SECURITY  CONSULTANT,  ENSPHERICS  (A  DIVISION  OF  CIBER) 

"As  an  IT  consultant,  1  continually  deal  with  many  people  from  different  disciplines  who  are  smarter  than  me.  Even 
so,  they  often  ask  me  which  books  I  rely  on  to  do  my  job.  The  Visible  Ops  book  just  became  one  of  them — in  my 
toolkit,  I'd  always  want  a  pocket  knife,  a  can  of  Sterno,  a  compass  and  this  guide." — RON  ZIKA,  ITSM 
CONSULTANT 
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Foreword 

I  remember  well  my  first  substantive  conversation  with  Gene  Kim  in  March  2003.  We  were  in  Orlando,  Florida, 
sitting  at  the  bar  after  a  full  day  of  mind-numbing  security  conference  presentations.  During  this  exchange,  I 
found  out  what  Gene  was  up  to,  and  I  had  one  of  those  proverbial  light  bulb  moments.  What  if  we  could  find 
a  way  to  define  mature  IT  operational  processes  and  then  embed  well-defined  security  controls  within  these 
processes?  If  we  could  do  this,  we  could  make  great  progress  in  addressing  security  in  the  normal  course  of 
operational  business,  instead  of  by  individual  heroics.  What  made  this  promising  and  exciting  was  that  Gene 
had  seen  this  in  action,  and  had  studied  how  certain  organizations  that  he  called  "best  in  class"  were  not  only 
doing  it,  but  doing  it  exceptionally  well. 

Gene  introduced  me  to  his  partner  in  crime,  Kevin  Behr.  We  found  that  we  had  similar  interests,  and  embarked 
on  finding  a  useful  way  to  work  together.  By  July  of  2003,  we  had  a  collaboration  agreement  in  place  between 
Carnegie  Mellon  University's  Software  Engineering  Institute  and  the  IT  Process  Institute.  In  October  2003,  we 
co-hosted  the  first  Best  in  Class  Security  and  Operations  Roundtable  at  the  SEI,  bringing  together  leaders  and 
high  performers  in  IT  operations  and  security. 

On  this  journey,  I  have  learned  the  following  from  Gene  and  Kevin: 

1.  They  have  a  unique  ability  to  observe,  analyze,  and  synthesize  information  and  experiences  from 
organizations  operating  across  a  wide  range  of  market  sectors.  In  doing  this  repeatedly,  they  have  created 
value,  resulting  in  strong,  long-term  relationships  of  trust  with  their  clients  and  partners. 

2.  They  have  identified  critical  characteristics  of  what  it  means  to  be  high  performing  in  IT  operations  and 
security,  as  evidenced  by  an  organization's  culture,  beliefs,  behaviors,  capabilities,  and  actions.  They  have 
observed  how  high  performing  organizations  view  the  problems  as  well  as  the  solutions.  This  handbook 
codifies  much  of  this  work. 

3.  They  passionately  believe  in,  and  have  begun  to  demonstrate,  the  power  of  mature  process  definitions  to 
bring  about  stability  and  control  in  complex  IT  environments,  including  the  requirement  for  auditable  and 
verifiable  controls. 

4.  They  have  invested  significant  time  and  energy  in  their  own  education  (and  mine)  and  in  building  a  rich 
value  network  of  leading  and  respected  professionals  in  IT  operations,  security,  and  audit  to  assist  and 
advise  this  work. 

5.  Their  observations  and  experiences  (and  those  of  their  clients  and  partners)  on  the  current  state  of  IT 
operations  and  security  are  remarkably  similar  to  those  of  the  software  development  community  before  the 
existence  of  a  body  of  community-accepted  software  development  process  definitions  (as  captured  in  the 
SETs  Capability  Maturity  Model®  for  Software). 

Why  has  the  SEI  embarked  on  this  journey  with  ITPI?  We  share  a  mutual  desire  to  improve  the  condition  of 
IT  operations  and  security.  These  capabilities  do  not  stand  alone;  they  live  in  an  enterprise  context.  The 
tougher  aspects  of  improvement  are  in  people  and  process,  even  though  the  community  at  large  still  tends  to 
view  localized  technology  solutions  as  the  path  for  improvement.  We  share  the  belief  that  sustained 
improvement  requires  the  creation  of  an  executive-level  community  of  practice,  who  will  integrate  the  goals 
and  objectives  of  IT  operations,  security,  audit,  risk  management,  process  management,  project  management, 
and  governance.  All  of  these  capabilities  are  required  to  bring  about  an  operational  environment  that  can 
deliver  repeatable,  predictable,  defined,  secure,  measurable,  and  measured  operational  processes,  thereby 
achieving  operational  excellence. 

We  share  the  objective  of  helping  organizations  make  common  sense  common  practice.  By  addressing  the 
difficult  questions,  "How  and  where  do  you  start?"  this  handbook  is  a  significant  step  in  the  right  direction. 

Julia  Allen 

Senior  Member  of  the  Technical  Staff 

Carnegie  Mellon  University,  Software  Engineering  Institute 

Networked  Systems  Survivability  Program,  home  of  the  CERT®  Coordination  Center 
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Introduction 

Practitioners  in  information  technology  (IT)  face  pressures  on  many  fronts.  In  addition  to  the  demands  to 
become  more  efficient,  IT  must  now  address  challenges  to  maintain  a  secure  state  and  comply  with  regulatory 
requirements.  For  example,  the  Sarbanes-Oxley  Act  of  2002  is  forcing  publicly  held  U.S.  corporations  to  attest 
to  the  fact  that  internal  controls  are  both  in  place  and  effective.  IT  operational  best  practices,  such  as  the 
Information  Technology  Infrastructure  Library  (ITIL),  provide  a  framework  to  start  defining  repeatable  and 
verifiable  IT  processes.  However,  as  organizations  attempt  to  use  ITIL  to  begin  their  journey  towards  process 
improvement,  they  face  two  very  difficult  questions:  How  and  where  do  you  start? 

This  handbook  provides  an  overview  of  the  methodology  that  we  have  developed  known  as  "Visible  Ops." 
Since  2000,  we  have  met  with  hundreds  of  IT  organizations  and  identified  eight  high-performing  IT  groups 
with  the  highest  service  levels,  best  security,  and  best  efficiencies.  What  was  most  amazing  about  them  was 
that  they  shared  the  following  attributes:  a  culture  of  change  management,  a  culture  of  causality,  and  a  culture 
that  fundamentally  valued  effective  and  auditable  controls,  promoting  fact-based  management.  Visible  Ops 
reflects  the  lessons  learned  about  how  these  organizations  work  and  describes  a  control-based  entry  point  into 
the  world  of  ITIL  that  others  can  leverage  to  springboard  their  own  process  improvement  efforts. 

In  the  IT  industry,  Stephen  Elliot,  an  IT  Senior  Analyst  with  IDC,  showed  that  on  average,  80%  of  IT  system 
outages  are  caused  by  operator  and  application  errors.1  This  motivated  our  need  to  dig  into  causal  factors  of 
infrastructure  downtime,  which  continually  revealed  shortfalls  in  change  management  practices.  Often,  many 
organizations  would  have  well-documented  change  management  practices,  but  in  reality,  no  one  ever  followed 
them.  In  many  of  these  cases,  the  goals  and  motivations  for  having  change  management  were  not  clear  to 
management  or  to  the  practitioners  themselves.  Another  key  finding  was  that  having  a  documented  change 
management  process  was  necessary,  but  far  from  sufficient,  to  achieve  high-performing  characteristics.  In  the 
high-performing  organizations  we  studied,  change  management  was  embedded  in  their  culture,  and  had  a  very 
different  meaning  than  in  typical  organizations.  This  book  is  dedicated  to  describing  those  practices  that  set 
the  high-performers  apart. 

Something  Must  Need  Improvement — Otherwise,  Why  Read  This? 

" The  most  likely  way  the  world  will  be  destroyed,  most  experts  agree,  is  by  accident.  That's  where  we  come 
in;  we're  computer  professionals.  We  cause  accidents."— Nathaniel  borenstein 

The  motivation  for  ITIL,  change  management,  and  overall  process  improvement  is  well  known.  The  trade  press 
is  full  of  stories  about  cost  cutting  measures,  outsourcing,  and  regulatory  requirements  from  Sarbanes-Oxley, 
HIPAA  (The  Health  Insurance  Portability  and  Accountability  Act  of  1996),  BASEL  II,  FISMA  and  so  forth.  The 
list  of  people  talking  about  the  problems  is  already  large  enough,  so  we  promise  to  keep  the  discussion  of  the 
problem  domain  to  a  minimum.  In  this  booklet,  the  issues  and  challenges  that  we  address  include: 


'  Source:  Stephen  Elliot,  Senior  Analyst  Network  and  Service  Management,  IDC,  2004.  Note,  additional  information  can  also  be  found 
from  the  Gartner  Group  at  http://www4.gartner.com/DisplayDocument?id=334197&ref=g_search 
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•  Organizations  have  change  management  processes,  but  view  these  processes  as  overly  bureaucratic  and 
diminishing  of  productivity.  There  must  be  more  to  change  management  than  bureaucracy,  good  intentions 
and  scarcely  attended  meetings. 

•  Organizations  where,  deep  down,  everyone  knows  that  people  circumvent  proper  processes  because  crippling 
outages,  finger-pointing,  and  phantom  changes  run  rampant. 

•  A  "cowboy  culture"  where  seemingly  "nimble”  behavior  has  promoted  destructive  side  effects.  The  sense  of 
agility  is  all  too  often  a  delusion. 

•  A  "pager  culture"  where  IT  operations  believes  that  true  control  simply  is  not  possible,  and  that  they  are 
doomed  to  an  endless  cycle  of  break/fix  triggered  by  a  pager  message  at  late  hours  of  the  night. 

•  An  environment  where  IT  operations  and  security  are  constantly  in  a  reactive  mode,  with  little  ability  to 
figure  out  how  to  free  themselves  from  fire-fighting  long  enough  to  invest  in  any  proactive  work. 

•  Organizations  where  both  internal  and  external  auditors  are  on  a  crusade  to  find  out  whether  proper  controls 
exist  and  to  push  madly  for  implementing  new  ones  where  they  are  not  in  place. 

•  Organizations  where  IT  understands  the  need  for  controls,  but  does  not  know  which  controls  are  needed  first. 

What  You  Do  And  Do  Not  Need  To  Know 

You  do  not  need  an  extensive  knowledge  of  ITU,  process  improvement,  security  or  audit  to  benefit  from  this 
book.  These  topics  are  introduced  in  this  handbook  as  they  become  necessary  in  the  Visible  Ops  methodology. 
Our  intent  is  to  create  a  working  knowledge  of  critical  concepts  in  these  domains,  both  to  serve  as  a  primer 
and  to  introduce  the  language  necessary  to  work  with  other  functional  departments,  such  as  security  and 
audit.  However,  we  recognize  that  each  one  of  these  domains  is  an  entire  vocation  and  field  of  expertise  unto 
itself,  so  we  list  recommended  resources  in  the  appendices  for  those  wishing  to  learn  more.  An  evolving  list  of 
resources  can  be  found  on  the  ITPI  Web  site  at  http://www.itpi.org. 

Structure  Of  The  Book 

This  booklet  presents  information  in  the  following  order: 

•  Visible  Ops:  What  is  it  and  why  does  it  work? 

•  There  are  four  Visible  Ops  phases.  In  each,  we  describe: 

-  Issues  and  indicators 

-  Specific  prescriptive  steps  to  solve  the  issues 

-  Benefits  and  what  you  are  likely  to  hear  as  the  steps  are  implemented 

•  Appendices  that  provide  a  brief  primer  on  auditable  controls,  information  on  how  to  proactively  prepare  for 
an  audit,  a  summary  of  ITIL,  and  other  helpful  resources. 

•  In  each  of  the  Visible  Ops  phases,  "Helpful  Tips  When  Preparing  for  an  Audit"  sections  appear  in  grey 
call-out  boxes,  highlighting  areas  of  special  interest  to  those  who  interact  with  auditors. 

Please  note  that  we  use  "production"  and  "operations"  interchangeably  to  specify  the  team  primarily  responsi¬ 
ble  for  day-to-day  infrastructure  operations  and  maintenance.  Specifically,  this  does  not  include  the  release 
management  team.  They  are  in  the  preproduction  portion  of  the  service  delivery  lifecycle. 
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Visible  Ops 

"It  is  not  enough  to  show  that  a  situation  is  bad;  it  is  also  necessary  to  be  reasonably  certain  that  the 
problem  has  been  properly  described,  fairly  certain  that  the  proposed  remedy  will  improve  it,  and  virtually 
certain  that  it  will  not  make  it  worse.”— Robert  conquest 

We  developed  the  Visible  Ops  methodology  because  everyone  seemed  to  be  asking  the  same  urgent  question: 
"1  believe  in  the  need  for  IT  process  improvement,  but  where  do  I  start?"  There  were  no  satisfactory  answers. 
Although  1TIL  provides  a  wealth  of  best  practices,  it  lacks  prescriptive  guidance:  In  what  order  and  how  should 
the  practices  be  implemented?  Moreover,  the  ITIL  books  remain  relatively  expensive  to  distribute  widely.  The 
third-party  information  that  is  publicly  available  on  ITIL  still  tends  to  be  too  general  and  vague  to  effectively 
aid  organizations.  This  booklet  provides  step-by-step  guidance  and  a  prescriptive  roadmap  for  organizations 
starting  or  continuing  their  IT  process  improvement  journey.  Visible  Ops  uses  ITIL  terminology,  and  is 
intended  to  be  an  "on-ramp"  to  the  rest  of  the  ITIL  body  of  knowledge. 

History  Of  Visible  Ops 

Since  early  2000,  Gene  Kim,  CTO  of  Tripwire,  Inc.,  and  Kevin  Behr,  CTO  of  IP  Services,  have  studied  what 
contributes  to  the  success  of  high-performing  IT  organizations.  IP  Services  is  a  business  process  outsourcing 
company,  managing  thousands  of  servers  for  Fortune  50  organizations.  At  IP  Services,  the  IT  operations  group 
reports  to  Kevin,  and  for  years,  he  tried  to  understand  how  to  best  increase  service  levels  and  decrease  cost  to 
maximize  value.  Tripwire  is  a  software  vendor  for  a  product  that  detects  change— it  was  originally  written  by 
Gene  in  1992  as  an  intrusion  detection  technology  to  help  system  administrators  recover  from  the  1988  Morris 
Internet  Worm.  Gene  has  spent  years  trying  to  understand  why  their  largest  customers  kept  insisting  that 
Tripwire's  software  was  not  a  security  technology,  but  instead,  a  technology  to  enforce  their  change 
management  processes. 

Kevin  and  Gene  began  working  together  when  they  discovered  they  had  a  common  passion  to  really 
understand  what  differentiated  high-performing  IT  organizations  from  their  more  typical  counterparts.  Visible 
Ops  began  to  take  shape  when  they  started  studying  a  list  of  organizations  that  Gene  had  been  keeping  for 
years,  which  he  called  "Gene's  list  of  people  with  amazing  kung  fu." 

After  years  of  research  and  investigation,  Kevin  and  Gene  now  refer  to  this  list  more  formally  as  "the  high 
performing  IT  operations  and  security  organizations  with  the  highest  service  levels,  as  measured  by  mean  time 
to  repair  (MTTR),  mean  time  between  failures  (MTBF),  and  availability,-2  the  early  integration  of  security 
requirements  into  the  operations  lifecycle;  the  lowest  amount  of  unplanned  work;  and  the  highest  server  to 
system-administrator  ratios."  What  makes  the  organizations  on  this  list  especially  astonishing  is  that  they  also 
have  more  efficient  cost  structures  than  lower-performing  organizations. 

To  coordinate  and  expand  their  efforts,  their  works  were  donated  to  the  Information  Technology  Process 
Institute  (ITPI).  The  ITPI  is  a  not-for-profit  organization  engaged  in  three  principle  areas  of  activity:  research, 
benchmarking  and  the  development  of  prescriptive  guidance  for  practitioners  and  business  executives.  The  ITPI 
has  collaboration  agreements  in  place  with  research  organizations  such  as  The  University  of  Oregon  Decision 
Sciences  program  and  The  Software  Engineering  Institute  at  Carnegie  Mellon  University.  The  ITPI  also  attracts 
many  other  contributors  through  the  ITPI  Community  of  Practice  List  (ICOPL).  At  the  time  of  writing,  there  are 


2  Appendix  D  is  the  glossary  of  terms. 
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hundreds  of  top  practitioners  from  IT  Operations,  Security,  Audit,  Management,  and  Governance  on  the  ICOPL, 
representing  thousands  of  years  of  IT  experience. 

Through  research,  development  and  benchmarking,  the  ITPI  creates  powerful  measurement  tools, 
prescriptive  adoption  methods  (such  as  Visible  Ops),  and  control  metrics  to  facilitate  management  by  fact. 
The  end  result  of  these  efforts  is  to  assist  organizations  with  their  IT  process  improvement  efforts.  This 
booklet  serves  as  an  example. 

Common  Characteristics  Of  High-Performing  IT  Organizations 

What  makes  high-performing  organizations  so  different  from  average  organizations,  both  qualitatively  and 
quantitatively?  We  observe  that  high-performing  IT  organizations  share  the  following  characteristics: 

•  Server  to  system  administrator  ratios  greater  than  100:1— This  means  that  each  system  administrator  controls 
more  than  100  servers.  In  contrast,  organizations  not  using  effective  processes  see  ratios  around  15:1. 
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Figure  1:  Server  to  System  Administrator  Ratio 
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•  Low  ratio  of  unplanned  to  planned  work— Only  5%  of  operational  expense  goes  toward  unplanned 
work.  From  our  ongoing  benchmarking,  we  find  that  average  organizations  spend  25-45%  of  their 
total  operational  expenses  on  unplanned,  unscheduled  work. 

•  Higher  staffing  early  in  the  IT  lifecycle— Continual  deployment  of  resources  and  staff  in  the 
preproduction  build  phase,  where  the  cost  of  defect  repair  is  least  expensive. 

•  Collaborative  working  relationships  between  functions— IT  operations  and  security  work  together  to 
solve  common  objectives,  with  IT  operations  performing  most  of  the  work  and  security  acting  as 
coach  and  consultant. 

•  Posture  of  compliance— Trusted  working  relationship  between  IT  operations  and  auditors,  because 
controls  are  visible,  verifiable  and  regularly  reported  on. 

•  Culture  of  change  management— Ubiquitous  understanding  throughout  the  organization  that 
changes  must  be  managed  in  order  to  achieve  business  objectives. 

•  Culture  of  causality — Through  the  use  of  controls  and  metrics,  these  groups  identify  and  solve 
problems  through  logical  use  of  cause  and  effect,  instead  of  a  culture  of  "let’s  see  if  this  works." 

•  Management  by  fact — These  organizations  value  controls  and  metrics,  not  only  to  aid  effective 
problem  solving,  but  to  aid  fact-driven  decision  making,  as  opposed  to  "management  by  belief"  or 
"management  by  the  honor  system." 

Why  Did  We  Use  ITIL? 

To  understand  what  the  best  in  class  organizations  were  doing,  Gene  and  Kevin  wanted  to  determine  the  union 
and  intersection  of  their  IT  processes.  In  other  words,  what  are  the  common  practices  of  all  the  high- 
performing  IT  operations  organizations  studied,  and  which  ones  are  necessary  to  achieve  the  high-performing 
characteristics?  Even  this  line  of  questioning  was  a  challenge,  because  each  organization  had  independently 
developed  their  own  processes,  and  each  had  Darwinistically  evolved  to  learn  from  past  mistakes  to  prevent 
certain  IT  disasters  from  ever  happening  again.3  Because  they  were  building  their  own  playbook,  as  opposed 
to  using  an  external  standard,  each  organization  called  similar  processes  by  different  names.  For  example,  one 
organization's  "change  management"  process  was  another's  "work  authorization  request  system"  or  "change 
control"  process.  As  a  result,  Kevin  and  Gene  first  needed  some  way  to  normalize  terminology  in  order  to 
determine  what  processes  these  organizations  had  in  common. 

To  resolve  this  terminology  problem,  they  did  a  Google  search  on  "release  management  and  change  manage¬ 
ment,"  which  brought  them  to  ITIL.  ITIL  is  a  compilation  of  IT  best  practices,  provided  without  prioritization 
or  any  prescriptive  structure.  ITIL  provides  a  framework  and  catalog  of  IT  operational  processes,  distilled  from 
thousands  of  man-years  of  experience.  Initially  created  in  the  late  1980s,  the  ITIL  body  of  knowledge  continues 
to  be  enhanced  and  better  organized,  most  significantly  (in  our  opinion)  in  the  form  of  the  BS  15000,  which 
divides  all  the  ITIL  disciplines  into  five  key  areas:  Release  Processes,  Control  Processes,  Resolution  Processes, 
Relationship  Processes,  and  Service  Delivery  Processes. 


3  Similarly,  FAA  insiders  say  that  "behind  every  regulation  is  an  airline  crash.” 
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Figure  2:  BS  15000  view  of  ITIL  process  areas 4 

The  BS  15000  categorizes  the  ITIL  capabilities  into  five  areas.  Each  are  briefly  described  below: 

•  Release  Process— This  process  area  answers  the  question  of  "where  does  infrastructure  come  from  before  it 
is  deployed?"  This  includes  activities  such  as  the  planning,  designing,  building,  and  configuring  of  hardware 
and  software.  Unfortunately,  release  processes  are  traditionally  the  last  process  area  that  organizations  invest 
in.  Yet  this  is  the  process  area  that  delivers  the  highest  return  on  investment,  because  it  encompasses  the 
entire  pre-production  infrastructure,  where  the  cost  of  defect  repair  is  lowest. 

•  Control  Processes — This  process  area  covers  maintaining  production  infrastructure,  not  only  to  prevent 
service  interruptions,  but  also  to  efficiently  deliver  IT  service.  This  is  done  through  change  management,  as 
well  as  asset  and  configuration  management.  BS  15000  defines  change  management  as  well  as  asset  and 
configuration  management  as  primary  controls.  As  Stephen  Katz,  former  CISO  of  Citibank,  once  said, 
"Controls  don't  slow  the  business  down;  like  brakes  on  a  car,  controls  allow  you  to  go  faster." 

•  Resolution  Processes — This  process  area  is  triggered  when  production  infrastructure  does  go  down,  service 
is  interrupted,  or  there  is  a  security  issue.  Incident  management  owns  the  customer  relationship,  and 
problem  management  owns  the  tasks  of  turning  each  problem  into  a  known  error  that  can  be  more 
efficiently  resolved  the  next  time,  it  happens.  All  too  often,  organizations  that  spend  too  much  time 
firefighting  are  unable  to  spend  time  in  the  previous  two  process  areas. 

•  Relationship  Processes— This  area  focuses  on  the  processes  necessary  to  support  effective  customer  relations 
as  well  as  the  management  of  third  party  vendors  from  a  performance  and  contractual  standpoint. 

•  Service  Delivery  Processes— The  goal  of  these  processes  is  to  provide  the  best  possible  service  levels  to  meet 
the  business  needs  of  the  organization.  This  process  area  includes  the  monitoring  and  management  of  IT 
infrastructure  as  it  relates  to  Security  Management,  Availability  and  Contingency  Management,  Capacity 
Management,  Financial  Management  and  Service  Level  Management  and  Reporting. 


1  BS  15000-1:2002— "IT  Service  Management:  Part  1:  Specification  of  Service  Management."  British  Standards  Institute.  September  2002. 
Page  2. 
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In  the  high-performing  organizations,  the  common  processes  were  in  the  release,  controls  and  resolution 
areas.  All  of  the  high-performers  had  repeatable  and  verifiable  processes  to  provision  infrastructure  in  a  known 
good  state.  They  had  a  culture  of  change  management  as  a  primary  way  to  do  work  and  they  all  used  causality 
in  their  problem  resolution  processes.  It  is  interesting  to  note  that  none  of  the  high-performing  shops  were 
using  ITIL  at  the  time  of  the  research.  But  again,  ITIL  provided  a  framework  to  name  and  normalize  the 
practices  that  the  high-performing  organizations  had  in  common. 

ITIL  is  still  not  in  a  form  where  you  can  simply  distribute  the  ITIL  volumes  to  your  entire  IT  organization  and 
expect  everyone  to  know  what  issues  to  tackle  first  and  what  everyone's  role  should  be.  Yet  experienced  IT 
practitioners  who  have  built  their  own  playbook  of  lessons  and  have  learned  from  their  own  disasters,  or  near¬ 
disasters,  are  likely  to  love  reading  the  ITIL  volumes.  They  will  see  reflections  of  their  own  belief  systems  and 
management  practices  in  the  ITIL,  and  recognize  the  wealth  of  hard-won  lessons  and  processes  contributed  by 
other  IT  practitioners  that  they  can  add  to  their  playbook.  With  these  expectations,  ITIL  can  be  a  tremendous 
wealth  of  useful  information.5 

One  last  note  on  ITIL:  We  are  continually  awed  and  amazed  that  so  many  organizations  have  re-created  the 
hard-won  lessons  embodied  in  ITIL  over  and  over  again.  Because  each  of  these  organizations  created  their  own 
methodology,  when  these  IT  operations  organizations  meet,  even  though  they  are  doing  very  similar  things, 
they  cannot  speak  a  common  language.  One  of  the  first  things  that  a  community  of  practice  must  develop  to 
share  best  practices  is  a  common  vocabulary.  By  using  ITIL,  we  normalized  the  various  terms  into  a  standard 
framework. 

In  our  opinion,  just  mapping  your  IT  operational  processes  to  ITIL  has  value.  It  allows  organizations  to  share 
best  practices  plus  leverage  the  tremendous  wealth  of  ITIL  and  its  various  advocacy  groups,  such  as  the  itSMF.6 
At  an  even  more  practical  level,  being  aware  of  ITIL  terminology  facilitates  interaction  with  other  IT 
organizations  and  lowers  the  risk  of  misunderstandings. 

Why  Visible  Ops  Works 

Since  2002,  we  have  presented  our  research  and  the  Visible  Ops  methodology  to  a  wide  cross-section  of  the  IT 
community.  Through  this  process,  we  have  received  positive  feedback  from  hundreds  of  people  in  virtually 
every  industry,  company  size  and  functional  role.  Sometimes  we  ask  ourselves  why  Visible  Ops  resonates  so 
well.  We  now  believe  that  it  is  because  Visible  Ops  is  both  logical  and  intuitive,  equally  accessible  to  technical 
and  non-technical  stakeholders.  Typical  reactions  are:  "This  makes  so  much  sense— I  have  to  show  this  to  my 
boss"  and  "Wow,  our  company  needs  to  do  this"  and  even  "Visible  Ops  shows  that  common  sense  is  rarely 
common  practice." 

By  replicating  how  the  high-performing  IT  organizations  work,  Visible  Ops  presents  practices  that  not  only 
make  sense,  but  also  can  be  implemented  in  any  organization  (i.e.  "this  really  isn't  rocket  science").  For  novice 
organizations,  Visible  Ops  provides  useful  guidance  on  where  to  start  their  improvement  efforts.  For  more 
mature  organizations,  Visible  Ops  provides  a  framework  for  continual  improvement. 


5  For  more  information  on  ITIL,  please  see  Appendix  B. 

6  http://www.itsmf.com/ 
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Visible  Ops  is  also  accessible  to  business  management,  security,  and  audit  because  it  is  controls-based.  By  being 
based  on  controls,  not  only  are  regulatory  issues  addressed,  but  controls  help  provide  the  reliable  delivery  of 
IT  service.  Visible  Ops  identifies  key  issues  that  undermine  service  levels  and  security,  and  provides  prescriptive 
guidance  to  address  them.  These  issues  are: 


Figure  3:  2004  IDC  Study  on  Causes  of  Network  Downtime 7 


•  Human  factors  effect  successful  change— Implementing  a  change  management  process  and  having  it 
actually  followed  are  two  very  different  things.  To  meet  the  requirements  of  the  business,  effective  change 
management  is  a  necessity.  In  order  for  it  to  work,  human  factors  must  be  addressed. 

•  80%  of  outages  are  self-inflicted — Donna  Scott,  VP  &  Research  Director,  Gartner,  notes  that,  "80  percent  of 
unplanned  downtime  is  caused  by  people  and  process  issues,  including  poor  change  management  practices, 
while  the  remainder  is  caused  by  technology  failures  and  disasters."7 8 

•  80%  of  MTTR  is  often  wasted  on  non-productive  activities— Determining  the  cause  of  an  outage 
consumes  a  great  deal  of  valuable  time  without  effective  change  management.  This  protracts  the  outage  and 
makes  repair  more  difficult. 

•  Absence  of  a  "culture  of  causality"— People  often  manage  and  work  by  intuition  and  "gut  feel." 
Consequently,  they  fail  to  use  problem-solving  skills  and  causality  to  resolve  issues.  The  Microsoft 
Operations  Framework  (MOF)  study  showed  that  their  high-performing  customers  reboot  servers  20  times 
less  often  than  average  and  have  five  times  fewer  "blue  screens  of  death." 

•  Rebuild  vs.  Repair— High-performing  organizations  make  it  easier  to  rebuild  infrastructure  than  to  repair  it. 
The  results  are  higher  and  more  predictable  service  levels,  plus,  by  rebuilding  from  documented  standard 
builds,  more  junior  staff  can  handle  repairs. 


7  Source:  Stephen  Elliot,  Senior  Analyst  Network  and  Service  Management,  IDC,  2004. 

8  Miller,  David.  "Hardware  High-Availability  Programs  in  Action.  (Product  Information)."  ENT  News.  June  1999. 
http://www.entmag.com/archives/article.asp?EditorialslD=6753 
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Visible  Ops — Key  Characteristics 

Visible  Ops  is  neither  a  death  march  nor  a  monumental  multi-year  undertaking.  In  fact,  we  have  seen 
organizations  successfully  complete  the  first  three  phases  of  Visible  Ops  in  90  days.  The  initial  part  of  the 
methodology  is  broken  down  into  manageable  sub-projects  prior  to  moving  into  a  continuous  improvement 
process.  The  goal  is  to  create  the  fewest  processes  necessary  to  enable  sustaining  improvement.  To  do  this,  each 
of  these  sub-projects  has  the  following  characteristics: 

•  Definitive  Projects— Each  phase  is  a  project  with  a  clearly  defined  objective. 

•  Ordered— Each  phase  is  specifically  designed  to  build  upon  the  previous  phase. 

•  Catalytic— Each  phase  returns  more  resources  to  the  organization  than  it  consumed,  thus  fueling  the  next 
phase. 

•  Auditable— Each  phase  creates  auditable  processes  that  generate  on-going  documentation  in  order  to  prove 
controls  are  working  and  effective. 

•  Sustaining— Each  phase  creates  enough  value  to  the  organization  that  the  processes  developed  remain  in 
place,  even  if  the  initial  driving  forces  behind  its  implementation  disappear. 

This  approach  has  many  benefits.  First,  because  of  the  relatively  short  length  of  each  phase,  concepts  and  their 
benefits  are  proven  faster.  Second,  getting  executive  sponsorship  and  funding  for  four  smaller  phases  is  easier 
than  for  a  big  vision  with  a  distant  promised  payoff. 

Facilitating  A  Productive  Working  Relationship  Between  IT  Operations,  Security 
And  Audit 

All  too  often,  IT  operations  groups  have  an  unproductive  working  relationship  with  security  and  audit.  Visible 
Ops  creates  a  framework  that  creates  productive  interfaces  between  these  groups,  through  repeatable,  verifiable 
and  auditable  IT  processes.  By  exposing  IT  controls  and  acceptance  points,  security  and  audit  are  able  to  review 
changes  before  they  are  implemented,  and  detect  when  these  controls  are  circumvented.  These  controls  are 
used  not  just  to  avoid  circumstances  which  can  lead  to  security  incidents  or  unplanned  work,  but  they  also 
allow  the  continual  monitoring  and  reduction  of  variance. 

Bill  Shinn,  a  System  Security  Engineer  with  a  Fortune  100  financial  institution,  has  studied  the  correlation 
between  the  amount  of  unplanned  work  and  the  number  of  security  incidents.  He  has  observed  that  as  the 
number  of  unplanned  changes  increases,  the  likelihood  of  insecure  configurations  increases  correspondingly, 
as  do  the  number  of  incidents  where  security  must  investigate  issues.  For  example,  security  may  be  called  upon 
during  a  network  outage  because  the  issue  is  obviously  "another  firewall  problem"  instead  of  an 
undocumented  change  made  by  the  network  administrators.  In  contrast,  when  changes  are  planned,  security 
has  a  chance  to  review,  approve  and  respond  to  the  changes  early  in  the  production  lifecycle  and  can  route 
issues  to  the  responsible  parties.  This  early  involvement  increases  the  overall  IT  organization's  ability  to  fix 
systemic  issues  that  lead  to  unnecessary  firefighting  and  security  problems. 

Similarly,  IT  auditors  often  are  exasperated  by  the  absence  of  documented  processes,  the  lack  of  a  defined 
desired  state,  and  an  inability  to  attest  to  whether  or  not  the  current  state  meets  the  documented  control 
objectives.  Without  these,  auditors  are  unable  to  determine  if  risks  and  controls  are  in  balance.  In  the  absence 
of  verifiable  controls,  they  must  go  into  "archaeology"  mode  and  make  a  judgment  on  whether  a  material  risk 
exists  or  not.  This  is  not  to  say  that  the  IT  operations  group  is  necessarily  doing  a  poor  job.  Indeed,  if  the  staff 
turnover  is  sufficiently  low,  the  "tribal  knowledge,"  or  combined  team  knowledge,  can  compensate  for  a  lack  of 
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formally  documented  processes,  observes  Ron  Zika,  a  Senior  Consultant  for  Waypoint,  Inc.  Visible  Ops  creates 
the  instrumentation  where  auditors  can  review  the  processes  and  controls  for  effectiveness  without  having  to 
enter  into  a  forensics  analysis  mode.  This  leads  to  a  more  productive  working  relationship,  smoother  audits 
and  less  time  spent  on  audit  preparation  and  remediation. 

Although  IT  operations,  security  and  audit  have  very  different  roles,  the  three  groups  are  often  needlessly  at 
odds  because  of  the  lack  of  effective  controls.  By  improving  processes  and  controls,  all  parties  benefit  by 
creating  a  more  productive  working  relationship  and  allowing  the  groups  to  more  efficiently  achieve  common 
business  objectives.  How  this  is  done  will  be  covered  in  more  depth  later  in  the  book. 

An  Overview  Of  The  Four  Visible  Ops  Phases 

Visible  Ops  gives  organizations  a  means  to  begin  their  process  improvement  journey.  After  studying  the  high- 
performing  organizations,  we  focused  the  methodology  on  four  key  phases: 
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Figure  4:  Visible  Ops'  Four  Phases  and  Relevant  ITIL  Process  Areas 


•  Phase  1:  "Stabilize  the  Patient"— In  this  phase,  we  curb  the  number  of  outages  by  freezing  change  outside 
of  scheduled  maintenance  windows.  We  also  modify  the  first  response  process  of  problem  managers  by 
ensuring  that  they  have  all  change  related  information  at  hand  about  what  could  have  caused  the  outage. 

•  Phase  2:  "Catch  &  Release"  and  "Find  Fragile  Artifacts"— Often,  infrastructure  exists  that  cannot  be 
repeatedly  replicated.  In  this  step,  we  inventory  assets,  configurations  and  services  to  identify  those  with  the 
lowest  change  success  rates,  highest  MTTR,  and  highest  business  downtime  costs.  Fragile  artifacts  are 
identified  and  then  treated  with  extra  caution  to  avert  risky  changes  and  massive  episodes  of  unplanned  work. 

•  Phase  3:  Establish  Repeatable  Build  Library— The  highest  return  on  investment  comes  from  implementing 
effective  release  management  processes.  This  step  creates  repeatable  builds  for  the  most  critical  assets  and 
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services  to  make  it  "cheaper  to  rebuild  than  to  repair."  We  take  the  priceless  paintings  identified  in  the 
previous  step  and  work  to  create  equally  functional  prints  that  can  be  mass-produced. 

•  Phase  4:  Enable  Continuous  Improvement— The  previous  steps  have  progressively  built  a  closed  loop 
between  the  release,  control  and  resolution  process  domains.  This  step  implements  metrics  to  enable  the 
continuous  improvement  of  all  of  these  process  areas  to  best  meet  business  objectives. 

Now  that  we've  provided  a  brief  overview,  let's  dive  into  the  details  of  Visible  Ops. 

Visible  Ops  In  Detail  0 

Visible  Ops  focuses  primarily  on  the  effective  management  of  change  to  begin  process  improvement  efforts. 
Organizations  have  two  means  to  embark  on  the  journey.  One  method  is  to  use  the  ITPI's  Integrity 
Management  Capabilities  Assessment  (IMCA)  to  identify  weak  areas  and  facilitate  implementation  planning 
(see  Appendix  C).  The  alternative  is  to  simply  follow  the  Visible  Ops  steps,  fully  completing  each  phase  before 
proceeding  to  the  next. 

Note  that  Visible  Ops  applies  to  all  infrastructure  systems  being  managed  by  IT  operations  spanning  servers,  |, 
databases,  routers,  switches,  firewalls,  networking  devices,  storage  systems  and  so  on.  Keep  in  mind  that  change 
affects  all  types  of  infrastructure— not  just  servers!  At  times  (such  as  in  the  third  phase)  Visible  Ops  may  seem  1 
focused  on  servers,  but  the  management  principles  are  broadly  applicable  to  all  types  of  IT  infrastructure. 
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Phase:  ©!  "Stabilize  The  Patient" 

And  "Modify  First  Response" 

Our  goal  in  this  phase  is  to  reduce  the  amount  of  unplanned  work  as  a  percentage  of  total  work  done  down  to  25% 
or  less.  Organizations  that  are  in  a  constant  firefighting  mode  can  have  this  percentage  at  65%  or  even  higher.  The 
first  phase  of  Visible  Ops  resembles  the  triage  system  used  by  hospitals  to  allocate  scarce  medical  resources.  In  a 
similar  fashion,  IT  must  identify  the  most  critical  systems  generating  the  most  unplanned  work  and  take 
appropriate  action  to  gain  control.  The  primary  goal  of  this  phase  is  to  stabilize  the  environment,  allowing  work 
to  shift  from  perpetual  firefighting  to  more  proactive  work  that  addresses  the  root  causes  of  problems. 

Issues  And  Indicators 


The  issues  and  symptoms  that  we  tackle  in  phase  one  are: 


Issue 

Narrative  Example 

Formal  service  levels  and/or 
informal  expectations  are  not 
being  met. 

"Despite  having  an  availability  target  for  99%  last  quarter,  we  did  not  achieve  it.  Truthfully, 
we  didn't  even  come  close.  Because  of  an  especially  horrendous  outage  that  spanned 
almost  two  days  on  December  1 8,  which  generated  all-nighters  for  everybody  and  lots  of 
expensive  overtime,  our  availability  figures  for  the  quarter  came  in  at  94%." 

IT  is  creating  the  majority  of 
their  own  work  through  self- 
inflicted  problems  related  to 
uncontrolled  change. 

Ik 

"Obviously,  94%  availability  is  not  acceptable.  We  looked  seriously  at  what  caused  this 
particular  outage,  so  we  could  prevent  it  from  happening  again.  Looking  back  at  those 
horrible  48  hours,  we  now  know  that  it  was  because  one  of  our  developers  decided  to 
upgrade  50%  of  the  Web  servers  with  some  new  code,  changing  93  critical  executables. 

It  caused  a  certain  Web  shopping  session  to  lock  up  the  servers.  The  upgraded  servers 
locked  up  so  hard  that  they  didn't  even  reboot — they  were  frozen  with  stack  traces 
everywhere.  Very  bad  news." 

When  systems  are  down,  80% 
of  the  MTTR  is  dominated  by 
simply  trying  to  characterize 
the  outage  and  determine 
causal  factors.  Only  20%  of 
the  recovery  time  is  spent 
actually  repairing  the 
infrastructure. 

"When  we  were  first  hit  by  these  failures,  our  entire  site  would  go  down,  taking  down  our 
whole  line  of  business.  We  convened  an  emergency  meeting  with  everyone  in  Ops  and 

R&D,  and  one  of  the  first  questions  we  asked  was,  'Did  anyone  change  anything?'  Of 
course,  the  answer  was,  'no.'  Everyone  swore  that  they  didn't  change  anything. 

"So,  while  the  business  was  losing  approximately  $20K  each  minute  because  we  were  in 
our  peak  holiday  retail  season,  we  went  from  pointing  fingers  at  each  other  to  eventually 
screaming  at  each  other.  Why  the  tension?  Because  this  had  happened  before.  No  one 
takes  accountability  for  changes,  and  without  proof,  all  we  had  to  go  on  was  suspicion. 

This  path  was  definitely  not  productive.  Quite  frankly,  because  these  disasters  had 
happened  before,  we  all  had  a  sense  of  deja  vu  and  lots  of  blame  was  being  cast  around." 
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Issue 


Narrative  Example 


When  changes  are  detected, 
who  made  the  change  and 
why  are  not  always  readily 
apparent.  Sometimes  changes 
have  a  very  long  fuse, 
detonating  long  after  the 
change  was  made. 


System  failures  happen  during 
very  inconvenient  periods, 
causing  stress  and  damaging 
IT's  reputation. 


Typically,  we  have  found 
that  average  organizations 
spend  35-45%  of  their 
time  on  unplanned  and 
unscheduled  work. 


It  is  all  too  easy  for  one 
change  to  undo  a  previous 
change  or  even  a  whole  series 
of  changes. 


Unnecessary  problem 
management  escalation  due 
to  invisible  failed  changes. 


"Cooler  heads  eventually  prevailed,  and  we  started  to  piece  together  some  useful  clues.  We 
discovered  that  there  was  a  development  upgrade  being  tentatively  explored.  It  was  nothing 
official,  but  it  was  a  promising  lead.  We  tracked  down  the  development  manager  leading  the 
effort,  who  told  us  that  the  responsible  developer  had  just  gone  on  vacation  the  previous  night. 

"This  was  terrible  news.  We  had  no  idea  whether  he  made  any  changes,  and  even  if  he  did, 
what  exactly  did  he  change?  How  could  we  best  unwind  the  changes  to  get  these  boxes 
back  to  a  running  state? 

"Eventually,  the  development  manager  caught  the  developer  at  some  airport  via  cell  phone, 
and  the  developer  admitted  that  he  had  done  a  'small  upgrade,'  but  was  adamant  that  his 
change  could  not  have  caused  the  outage.  He  said  that  it  was  'inconceivable'  that  his 
change  would  cause  the  failure — he  actually  used  the  word  'inconceivable!'  just  like  in  the 
movie  The  Princess  Bridel 

"'Inconceivable.'  Yeah,  right.  When  we  copied  these  supposedly  harmless  executables  onto  a 
test  box,  we  were  immediately  able  to  replicate  the  problem.  It  may  seem  bad  that  we 
chewed  up  over  24  hours  to  get  to  this  point,  but  then  the  real  bad  news  hit  us..." 

"Did  I  tell  you  that  this  outage  happened  four  shopping  days  before  Christmas?  And  that  our 
company  does  80%  of  our  business  in  December?  Did  I  tell  you  that  we  were  unable  to 
process  orders  on  what  should  have  been  the  busiest  revenue  day  for  our  online  Web 
commerce  systems?  Our  executives  were  so  furious  that  they  actually  brought  in  external 
auditors  to  figure  out  whose  butt  to  kick.  That  pretty  much  catches  us  up  to  the  current 
state  of  affairs. 

"Auditors.  Great.  Well,  I've  had  my  butt  chewed  out  before,  and  I've  survived  all  the  audits 
so  far.  But  I'm  pretty  sure  the  executives  brought  in  some  consultants  as  well,  to  'explore 
options.'  At  this  point,  everyone  fears  the  worst:  Outsourcing." 

"We've  had  over  ten  almost-catastrophic  failures  this  year.  There  may  be  good  reasons  that 
there  is  a  feeling  in  the  IT  operations  staff  that  things  are  in  utter  chaos.  For  whatever 
reason,  this  is  definitely  not  the  information  that  is  being  presented  to  management.  In  fact, 
I've  seen  some  of  the  IT  reports  being  presented  to  the  executives,  and  it  actually  shows  us 
doing  a  good  job:  99.9%  availability,  no  major  failures,  etc." 

"The  crazy  part  is  how  much  they're  focusing  on  the  wrong  numbers  to  come  up  with 
reports  that  say  what  a  great  job  we're  doing!  When  it  comes  down  to  it,  we  are  spending 
too  much  time  on  crisis  management  and  falling  behind  on  projects  the  rest  of  the  time. 
When  SQL  Slammer  hit  us  one  weekend,  organization-wide,  we  probably  spent  over  $35 
million  on  unplanned  work.  Because  it  was  a  weekend,  it  didn't  hit  the  availability  numbers, 
but  how  can  we  ignore  all  that  unplanned  work?" 

"So  you'd  think  that  after  the  SQL  Slammer  disaster,  we  would  have  learned  our  lesson, 
right?  I  wish  that  were  true.  Unfortunately,  after  all  the  emergency  patching,  in  the  next 
quarterly  build,  half  of  the  servers  that  we  deployed  didn't  have  one  of  the  critical  patches 
installed,  and  the  same  thing  happened  to  us  less  than  two  months  after  the  initial  SQL 
Slammer  attack.  We  looked  like  complete  goofballs,  and  frankly,  I'm  wondering  why 
someone  hasn't  been  canned  for  this!" 

"Not  only  are  we  spending  more  time  on  unplanned  work  than  planned  work,  the  majority 
of  the  unplanned  work  is  self-generated.  Everyone  is  so  bad  at  making  configuration 
changes  of  any  kind  that  I  look  forward  to  any  time  when  we  are  not  allowed  to  make 
changes.  In  fact,  our  developers  create  so  much  carnage  during  our  application  code 
upgrades  that  here  in  IT  operations,  we  sometimes  joke  about  crashing  the  developers' 
systems  so  we  can  get  some  real  work  done. 

"When  our  critical  app  servers  go  down,  the  first  thing  we  say  to  ourselves  is,  'Hey,  I  bet  a 
developer  just  did  a  code  push!"' 
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Issue 

Narrative  Example 

Due  to  low  change  success 
rates,  high  rates  of  change 
and  high  MTTR,  IT  is 
spending  all  of  their  time 
doing  unplanned  work. 

Overall,  there  is  a  lack  of 
confidence  in  IT. 

"But  the  problem  is,  the  development  folks  aren't  the  only  guilty  ones.  Many  of  the 
operational  changes  that  we  make  are  just  as  risky, 

"We  started  to  compute  our  change  success  rate,  and  while  development  only  bats  40%, 
our  change  success  rate  is  nothing  to  write  home  about  either.  In  the  last  month,  only  70% 
of  the  changes  we  made  worked  the  first  time  without  generating  a  firefighting  episode." 

"So,  not  only  are  we  all  ticked  off  with  the  performance,  but  we're  also  getting  our  butts 
kicked  by  senior  management  pretty  hard  these  days.  In  fact,  there  are  rumors  that  they  are 
talking  to  potential  outsourcers  now,  and  that  isn't  doing  morale  any  good." 

Stabilize  The  Patient 

"To  err  is  human.  To  really  screw  up  requires  the  root  password." — unknown 

The  first  goal  is  to  stabilize  the  patient.  We  need  to  decrease  the  amount  of  unplanned  work  in  order  to  free 
up  enough  resources  to  create  proactive  processes.  To  do  this,  we  start  where  the  most  damage  is  being  done. 
Fortunately,  this  happens  to  be  the  place  where  we  also  have  the  most  control.  If  80%  of  our  injuries  are  self- 
inflicted,  then  that  means  that  we  are  causing  80%  of  our  unplanned  work.  Therefore,  we  must  start  by 
reducing  the  number  of  self-inflicted  problems  by  gaining  control  of  the  change  process. 

Start  by  identifying  the  systems  and  business  processes  that  generate  the  greatest  amount  of  firefighting.  When 
problems  are  escalated  to  IT  operations,  which  servers,  networking  devices,  infrastructure  or  services  are 
constantly  being  revisited  each  week?  (Or  worse,  each  day!)  These  items  are  your  list  of  "most  fragile  patients” 
which  are  generating  the  most  unplanned  work.  These  are  the  patients  that  must  be  protected  from 
uncontrolled  changes,  both  to  curb  firefighting  and  to  free  up  enough  cycles  to  start  building  a  safer  and  more 
controlled  route  for  change. 

For  each  fragile  patient  (i.e.  server,  networking  device,  asset,  etc.),  do  the  following: 

1.  Reduce  or  Eliminate  Access:  Clear  everyone  away  from  the  asset  unless  they  are  formally  authorized  to 
make  changes.  Because  these  assets  have  low  change  success  rates,  we  must  reduce  the  number  of  times  the 
dice  are  rolled. 

2.  Document  the  New  Change  Policy:  Our  recommended  change  policy  is  very  simple:  "Absolutely  no 
changes  to  this  asset  unless  authorized  by  me."  This  policy  is  our  preventive  control  and  creates  an 
expectation  of  behavior. 

3.  Notify  Stakeholders:  After  the  initial  change  policy  is  established,  notify  all  of  the  stakeholders  about  the 
new  process.  Make  sure  the  entire  staff  sees  it:  email  it  to  the  team,  print  it  out,  and  add  it  to  login  banners. 

14.  Create  Change  Windows:  Work  with  stakeholders  to  identify  periods  of  time  when  changes  to  production 
systems  can  be  made.  Our  goal  will  be  to  eventually  schedule  all  changes  into  these  maintenance  windows. 
Amend  the  change  policy  accordingly.  For  example,  "Once  I  authorize  the  changes,  1  will  schedule  the  work 
to  be  performed  during  one  of  the  defined  maintenance  windows  on  either  Saturday  or  Sunday  between  3 
and  5  pm." 
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5.  Reinforce  the  Process:  By  now,  you  have  defined  a  clear  expectation  of  how  changes  are  to  be  made.  Make 
sure  that  people  are  aware  of  the  new  process  and  reinforce  it  constantly.  For  example,  "Team,  let  me  be  clear 
on  this:  These  processes  are  here  to  enable  the  success  of  the  entire  team,  not  just  individuals.  Anyone  making 
a  change  without  getting  authorization  undermines  the  success  of  the  team,  and  we'll  have  to  deal  with  that. 
At  a  minimum,  you'll  have  to  explain  why  you  made  your  cowboy  change  to  the  entire  team.  If  it  keeps 
happening,  you  may  get  the  day  off,  and  eventually,  it  may  prevent  you  from  being  a  part  of  this  team." 

Electrify  The  Fence 

"What  is  often  overlooked  is  that  if  one  person  can  single-handedly  save  the  ship,  that  one  person  can 
probably  single-handedly  sink  the  ship,  too."— unknown 

In  the  previous  step,  we  have  specified  how  and  when  changes  can  be  made.  This  is  the  first  preventive  change 
process  and  policy.  In  reality,  our  experience  has  shown  that  merely  specifying  the  correct  way  to  make  changes 
rarely  results  in  everyone  adhering  to  the  process.  We've  found  that  managing  change  on  the  honor  system  is 
not  enough,  for  a  variety  of  reasons.  Sometimes  IT  staff  may  be  unwilling  to  change  behaviors  (e.g.  "they're  the 
IT  cowboys  who  refuse  to  stop  shooting  from  the  hip"),  development  staff  may  not  communicate  well  with  IT 
operations  (e.g.  "they're  the  same  developers  who  do  not  even  show  up  to  meetings"),  or  people  may  just  make 
mistakes  (e.g.  "our  best  engineer  made  a  seemingly  trivial  unauthorized  change  that  just  blew  up"). 

So  far,  we  have  put  a  fence  around  the  systems  where  unauthorized  changes  were  causing  the  most  carnage. 
In  this  step,  we  will  electrify  the  fence.  We  do  this  to  keep  everyone  accountable  and  responsible  for  playing 
by  the  rules.  Our  goal  is  to  start  creating  a  culture  of  change  management.  To  do  this,  proper  change 
monitoring  must  be  in  place  so  we  can  "trust,  but  verify."  We  will  use  this  instrumentation  to  detect  and  verify 
that  changes  are  happening  within  the  specified  change  management  process,  and  also  to  negatively  reinforce 
and  deter  changes  that  are  not. 

We  must  be  aware  of  changes  on  all  infrastructure  that  we  are  managing:  servers,  routers,  network  devices, 
databases,  and  so  forth.  Each  detected  change  must  either  map  to  authorized  work,  or  it  must  be  flagged  for 
investigation.  Critical  questions  that  need  to  be  answered  are: 

•  Who  made  the  change? 

•  What  did  they  change? 

•  Should  it  be  rolled  back?  If  so,  then  how? 

•  How  do  we  prevent  it  from  happening  again  in  the  future? 

Answering  these  questions  forms  one  of  the  primary  goals  of  the  investigation  process.  Many  organizations 
start  the  formal  investigation  by  sending  an  email  to  the  entire  team  describing  the  unauthorized  change,  and 
give  the  team  a  fixed  time  (e.g.  four  hours)  for  someone  to  step  forward  and  explain  why  they  circumvented 
the  change  process. 

Although  you  can  audit  changes  manually,  change  monitoring  and  reporting  software,  such  as  Tripwire®, 
automates  the  detection  and  reporting  of  changes.  We  recommend  scanning  systems  for  changes  at  least  daily 
or  after  each  maintenance  window,  whichever  is  more  frequent.  Almost  universally,  people  implementing  this 
phase  are  surprised  and  alarmed  to  see  how  many  changes  are  being  made  "under  the  radar."  Some 
organizations  care  so  much  about  controlling  unauthorized  change  that  they  have  an  end-of-shift  audit 
process,  where  a  change  report  is  generated  at  the  end  of  each  shift,  and  the  operations  manager  is  required 
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to  attest  that  all  changes  can  be  mapped  to  authorized  work  or  have  been  rolled  back.  This  way,  managers  are 
held  accountable  for  changes  made  on  their  watch! 

Because  we  are  now  detecting  changes  that  circumvent  the  change  authorization  process,  we  begin  to  "manage 
by  fact"  rather  than  "manage  by  belief."  We  no  longer  rely  on  verbal  promises  or  assurances  of  good  behavior. 
When  a  service  fails,  a  server  crashes,  or  an  issue  has  been  escalated  to  problem  managers,  we  can  generate  and 
scrutinize  reports  of  actual  system  changes.  When  stakeholders  start  seeing  how  often  these  issues  are  caused 
by  failed  change,  a  culture  of  change  management  starts  to  emerge. 

The  key  to  creating  a  successful  culture  of  change  management  is  accountability.  If  the  change  process  is 
repeatedly  bypassed,  management  must  be  willing  to  take  appropriate  disciplinary  action,  which  may  range 
from  further  training,  public  shaming,  and  eventually  to  formal  HR-related  measures. 

One  last  comment  on  the  importance  of  detective  controls:  Monitoring  changes  gives  us  a  critical  safety 
mechanism,  just  like  a  rock  climber  with  a  ratchet.  The  ratchet  allows  the  rope  to  move  in  one  direction, 
preventing  the  climber  from  falling.  Monitoring  change  to  enforce  the  process  prevents  our  organization  from 
sliding  back  into  a  state  of  uncontrolled  change. 

Modify  First  Response:  The  Catalytic  Key 

"Grant  me  the  Serenity  to  accept  the  things  I  can  not  change,  Courage  to  change  the  things  I  can,  and 
Wisdom  to  know  the  difference."— dr.  reinhold  niebuhr  (excerpted  from  the  serenity  prayer) 

In  high-performing  IT  organizations,  the  change  management  process  is  catalytic,  returning  obvious  and 
measurable  value  back  to  the  organization  daily.  In  this  step,  we  will  make  our  change  process  catalytic,  to 
ensure  that  the  organizations  not  only  see  the  benefit,  but  also  internalize  and  perpetuate  the  process.  Despite 
all  of  our  research  findings  to  the  contrary,  change  management  is  often  viewed  as  a  burdensome  bureaucracy 
that  consumes  resources,  time,  money,  and  spiritual  energy.  To  prevent  this,  we  must  replicate  how  the  high- 
performing  organizations  use  their  change  process.  They  integrate  their  change  management  processes  with 
the  problem  resolution  processes  to  drastically  reduce  MTTR.  Specifically,  when  service  outages  occur,  high- 
performing  organizations  first  look  at  all  approved  and  detected  changes  before  making  a  diagnosis. 

Why  do  they  do  this?  Recall  that  80%  of  all  outages  are  caused  by  change,  and  that  80%  of  MTTR  is  spent 
trying  to  determine  what  changed.  High-performing  IT  organizations  eliminate  change  as  a  causal  factor  for 
an  outage  as  early  as  possible  in  the  repair  cycle.  They  identify  the  assets  directly  involved  in  the  service 
outage,  and  examine  all  changes  made  on  those  assets  in  the  previous  72  hours.  This  information  is  then  put 
into  the  work  ticket,  as  well  as  the  list  of  all  authorized  and  scheduled  changes.  By  doing  this,  when  issues  are 
escalated  to  problem  managers,  they  have  all  relevant  and  causal  evidence  already  at  hand.  Typically,  when 
equipped  in  this  way,  problems  managers  can  successfully  diagnose  issues  without  logging  into  any 
infrastructure  over  50%  of  the  time! 

If  no  changes  were  authorized  for  the  specific  asset  and  no  changes  were  detected,  then  the  investigation  circle 
is  widened  to  the  next  ring  of  infrastructure  that  supports  the  affected  asset.  Again,  we  do  this  by  examining 
our  change  management  records  and  change  monitoring  systems,  not  by  logging  into  infrastructure.  For 
example,  if  the  database  service  experiences  an  outage,  we  start  our  investigation  by  looking  for  authorized, 
scheduled  and  detected  changes  on  the  database  server.  If  none  are  found,  we  then  search  for  authorized, 
scheduled  or  detected  changes  on  systems  that  support  the  database  service,  such  as  the  operating  systems, 
supporting  networking  devices,  and  the  other  dependencies.  We  have  found  that  over  70%  of  service-affecting 
issues  can  be  resolved  in  this  manner. 
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Recall  that  the  Microsoft  Operations  Framework  (MOF)  study  showed  that  their  best  customers  with  the 
highest  service  levels  rebooted  their  servers  20  times  less  often  than  average.  This  is  because  they  manage 
problems  by  using  causality  and  solving  root  causes,  as  opposed  to  "rebooting  the  server  to  see  if  the  problem 
goes  away."  By  integrating  problem  management  with  our  change  management  processes,  we  will  facilitate  this 
same  type  of  desirable  behavior,  and  furthermore,  change  management  becomes  catalytic.  IT  operations  will 
see  value  in  knowing  when  changes  are  supposed  to  happen,  and  being  aware  of  what  changes  actually 
occurred.  In  fact,  many  organizations  have  seen  such  value  in  this  process  that  they  complain  loudly  when 
this  "instrumentation"  is  taken  away— no  one  likes  flying  blind. 

Helpful  tips  include: 

•  When  creating  the  incident  in  the  trouble-ticketing  or  change  workflow  system,  pre-populate  the  ticket  with 
all  detected  changes,  as  well  as  any  other  authorized  changes  made  in  the  last  30,  60,  and  90  days. 

•  If  you  do  not  find  changes  in  the  asset  in  question,  increase  the  search  radius  to  include  the  next  ring  of 
dependent  infrastructure. 

Create  The  Change  Team 

In  the  previous  steps,  we  have  started  to  specify  the  correct  path  for  change  and  built  the  mechanisms  to 
ensure  that  the  process  is  being  followed.  In  this  next  step,  we  will  continue  to  develop  the  change 
management  process  by  creating  a  Change  Advisory  Board  (CAB),  comprised  of  the  relevant  stakeholders  of 
each  critical  IT  service.  These  stakeholders  are  the  people  who  can  best  make  decisions  about  changes  because 
of  their  understanding  of  the  business  goals,  as  well  as  technical  and  operational  risks.  Kurt  Spence  from  HP 
states,  "All  business  decisions  result  in  an  IT  change  event  of  some  kind."  Our  goal  is  to  make  sure  that  they 
are  fact-based  decisions,  resulting  in  managed  changes. 

Common  stakeholders  on  the  CAB  often  include  the  following  people  and  roles: 

•  VP  of  Operations— is  ultimately  accountable  for  availability  and  has  final  authority  on  change  approval 

•  Director  of  Network  Operations— reviews  priorities  and  impacts  on  resources 

•  Security  Lead— reviews  changes  for  security  implications 

•  Ops  Systems  Engineering  Lead— reviews  changes  for  pre-production  implications 

•  Service  Desk  Manager— reviews  changes  for  customer-facing  implications 

•  Internal  Audit— may  attend  to  better  understand  how  changes  are  approved 

One  mistake  organizations  make  is  that  they  believe  urgent  changes  (i.e.  emergency  changes)  can  be  handled 
outside  the  CAB  meetings.  This  assumption  is  false!  Consider  that  virtually  all  cowboy  organizations  believe 
that  their  changes  are  both  urgent  and  safe.  In  reality,  emergency  changes  are  the  most  critical  to  scrutinize  and 
are  the  changes  that  require  the  most  deliberation  to  approve.  For  these  types  of  changes,  create  an  emergency 
change  process  with  a  defined  CAB  emergency  committee  (CAB/EC)  who  can  assemble  quickly  to  review  these 
requests.  All  changes  that  create  risks  must  be  evaluated  and  authorized,  especially  during  emergencies. 
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Create  A  Change  Request  Tracking  System 

A  prerequisite  for  any  effective  change  management  process  is  the  ability  to  track  requests  for  changes  (RFCs) 
through  the  authorization,  implementation,  and  verification  processes.  Paper-based  manual  systems  quickly 
become  impractical  when  the  organization  is  large,  or  complex,  or  when  the  number  of  changes  is  high.  Because 
of  this,  most  groups  use  some  computerized  means  to  track  RFCs  and  assign  work  order  numbers.  Some  refer  to 
these  applications  as  "ticketing  systems"  or  "change  workflow  systems."  Examples  of  systems  include  HP  Service 
Desk,  Remedy  ARS  (Action  Request  System)  and  Best  Practical  RT/RTIR. 

The  primary  goals  of  a  change  request  tracking  system  are  to  document  and  track  changes  through  their 
lifecycle  and  to  automate  the  authorization  process.  Secondarily,  the  system  can  generate  reports  with  metrics 
for  later  analysis.  Each  change  requester  should  gather  all  the  information  the  change  manager  needs  to  decide 
whether  the  change  should  be  approved.  In  general,  the  more  risky  the  proposed  change  is,  the  more 
information  that  is  required.  For  instance,  a  business  as  usual  (BAU)  change,  such  as  rebooting  a  server  or 
rotating  a  log  file,  may  require  very  little  data  and  oversight  prior  to  approval.  On  the  other  hand,  a  high-risk 
change  such  as  applying  a  large  and  complex  security  patch  on  a  critical  production  server  may  not  only 
require  good  documentation  of  the  proposed  change,  but  also  extensive  testing  before  it  can  even  be 
considered  for  authorized  deployment. 

Start  Weekly  Change  Management  Meetings  (To  Authorize  Change) 

And  Daily  Change  Briefings  (To  Announce  Changes) 

"If  it  is  too  complicated  to  understand,  it  is  too  complicated  to  govern."—' tom  horton 

Now  that  we  have  identified  the  change  stakeholders  by  creating  the  CAB,  the  next  step  is  to  create  a  forum 
for  them  to  make  decisions  on  requested  changes.  The  CAB  will  authorize,  deny,  or  negotiate  a  change  with 
the  requester.  Authorized  changes  will  be  scheduled,  implemented,  and  finally  verified.  The  goal  is  to  create  a 
process  that  enables  the  highest  successful  change  throughput  for  the  organization  with  the  least  amount  of 
bureaucracy  possible.  While  they  may  seem  unnatural  at  first,  with  practice,  weekly  15  minute  change 
management  meetings  are  possible.  Take  special  care  to  avoid  an  attitude  of  "just  get  it  done,"  which  allows 
people  to  make  changes  that  circumvent  the  change  approval  process.  If  we  make  it  easy  for  all  changes  to  flow 
through  our  process,  it  will  soon  be  easier  to  use  the  process  than  to  circumvent  it,  even  during  emergencies. 

CABs  must  meet  on  a  regular  published  schedule  that  all  stakeholders  understand.  To  start,  we  will  have  each 
CAB  meet  weekly.  The  agenda  will  begin  with  recording  attendance  and  progress  to  the  following: 

Deal  With  Old  Business: 

1.  Review  any  failed  changes  or  change  management  circumventions,  as  these  are  likely  to  have  consequences. 

2.  Review  and  close  action  items  from  the  previous  meeting  minutes. 

3.  Discuss  any  problems  resulting  from  old  changes.  If  necessary  set  up  a  Post-Implementation  Review 
meeting  to  deal  with  issues. 

4.  Review  any  category  of  changes  that  should  be  categorized  as  "business  as  usual"  to  avoid  repeated 
examination  by  the  CAB. 
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Deal  With  New  Business 

1.  Review  the  list  of  requests  for  change  (RFC),  and  agree  upon  an  order  in  which  to  evaluate  them. 

2.  Examine  RFCs  for  risks  of  collisions  or  interference,  due  to  proximity  of  simultaneous  or  similar  changes: 

a.  Group  changes  by  category  (software,  hardware,  server,  etc.). 

b.  Examine  proposed  change  dates  for  collisions  and  or  potentially  incompatible  activities.  (For  example, 
multiple  OS  upgrades  on  the  same  day.) 

3.  Find  any  RFCs  which  are  not  actually  tasks,  but  projects.  Projects,  unlike  tasks,  have  multiple  steps  that  are 
dependent  upon  each  other.  Verify  that  a  project  manager  has  been  assigned  for  planning,  coordination 
and  execution.  If  the  project  dependencies  have  not  been  adequately  evaluated,  it  is  entirely  appropriate  to 
reject  the  RFC  and  request  that  new  RFCs  be  submitted  for  each  of  the  tasks. 

4.  Evaluate  and  authorize  the  RFC. 

5.  Schedule  the  approved  changes  on  the  Forward  Schedule  of  Change  (FSC)  and  assign  a  change 
implementer.  (People  with  project  management  experience  are  great  for  coordinating  these  activities.) 

6.  Send  rejected  RFCs  back  to  the  respective  change  requester  for  further  clarification  or  response  to 
CAB  comments. 

The  CAB  meeting  should  be  restricted  to  evaluating  and  approving  RFCs,  and  should  not  get  bogged  down  in 
process  issues.  Instead,  dock  these  issues  to  be  handled  in  a  separate  meeting.  The  goal  is  to  keep  the  change 
management  meetings  focused  on  accomplishing  the  task  at  hand:  management  of  change.  When  an  RFC  is 
rejected,  the  change  requester  should  respond  by  addressing  the  concerns  or  providing  more  information. 
Remember  that  the  function  of  the  CAB  is  to  identify  which  changes  are  risky,  not  to  come  up  with  solutions— 
doing  this  takes  too  much  time.  With  practice,  a  CAB  meeting  can  be  finished  in  15  minutes. 

The  following  are  typical  questions  to  ask  when  evaluating  a  change  for  authorization.  Not  all  changes  require 
answers  to  all  of  these  questions,  but  as  the  risk  increases,  insist  on  having  good  answers  to  more  of  these  questions: 

•  "Who”  Questions 

-  Who  will  be  affected  by  the  change?  Ensure  that  there  is  appropriate  representation  on  the  CAB  to 
make  decisions. 

-  Who  could  be  affected  by  the  change  if  it  fails? 

-  Who  from  the  potentially  affected  group(s)  has  signed  off  on  the  change? 

-  Who  is  performing  the  change  (the  "change  builder")? 

-  Who  has  reviewed  the  proposed  change? 

-  Who  is  driving  the  change  (the  "change  owner")? 

-  Who  is  the  project  manager  if  this  change  involves  more  than  one  step? 
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•  "What"  Questions 

-  What  assets  are  the  targets  of  the  proposed  change? 

-  What  is  the  change  timeline? 

-  What  is  the  change  review  priority  based  on  the  associated  risk  and  urgency? 

-  Urgent— This  change  could  cause  a  loss  of  service  or  severe  impairment  of  usability  to  a  large 
percentage  of  users  or  a  mission  critical  business  system  and  is  needed  right  away.  Immediate  action 
is  required  and  an  urgent  CAB  or  CAB/EC  meeting  may  need  to  be  scheduled. 

-  High— This  change  could  severely  impact  a  large  number  of  users.  This  change  should  be  given  the 
highest  priority  for  change  planning,  building,  testing  and  implementation  in  order  to  meet  the  next 
available  maintenance  window. 

-  Medium— The  impact  of  this  change  is  not  large,  but  can  not  be  postponed  until  the  next  scheduled 
release  or  upgrade  window. 

-  Low— The  change  is  important,  but  has  relatively  low  risk  and  can  occur  during  the  next  scheduled 
release  or  maintenance  window. 

-  What  assets  or  processes  depend  on  the  targeted  assets? 

-  What  will  the  successful  change  look  like  when  implemented? 

-  What  business  processes  need  to  be  verified  after  making  the  change? 

-  What  is  the  business  or  technical  reason  for  the  change? 

-  What  will  happen  if  the  change  is  not  made? 

•  "When"  Questions 

-  When  will  the  change  be  performed? 

-  When  will  it  be  finished? 

-  When  will  the  benefits  of  the  change  be  realized? 

•  "How"  Questions 

-  How  will  the  change  be  implemented  (in  waves,  one  at  a  time,  etc.)? 

-  How  will  we  verify  success? 

-  How  will  issues  be  escalated? 

-  How  successful  were  similar  changes  in  the  past?  (i.e.  change  success  rate) 

•  "What  if"  Questions 

-  What  is  the  rollback  plan  if  the  change  should  fail  for  some  reason? 

-  What  is  the  worst  possible  outcome  associated  with  this  change? 

-  What  will  the  worst  case  service  outage  be? 

Again,  not  all  changes  will  require  the  same  level  of  scrutiny.  Business  as  usual  (BAU)  changes  which  are 
known,  regularly  executed,  and  have  a  low  risk  do  not  need  such  detail.  Conversely,  any  change  that  is  new 
or  perceived  as  having  material  risk  must  have  more  detail  to  allow  for  accurate  risk  assessment  by  the  CAB. 


PHASE  ONE 


27 


Miscellaneous  Change  Management  Do's  And  Don'ts 

"It’s  not  the  strongest  species  that  survive,  nor  the  most  intelligent...  but  the  one  most  responsive  to 
change."— Charles  darwin 

Here  are  some  tips  for  change  management. 

Items  to  do: 

•  Do  post-implementation  reviews  to  determine  whether  the  change  succeeded  or  not 

•  Do  track  the  change  success  rate 

•  Do  use  the  change  success  rate  to  learn  and  avoid  making  historically  risky  changes 

•  Do  make  sure  everyone  attends  the  meetings,  otherwise  auditors  have  a  good  case  that  this  is  a  non¬ 
functioning  control 

•  Do  categorize  the  disposition  of  all  changes.  In  other  words,  all  outcomes  must  be  documented  once  a 
change  is  approved.  Three  potential  outcomes  are: 

-  Change  Withdrawn— the  change  requester  rescinds  the  change  request  along  with  the  reason  why.  This 
should  not  be  flagged  as  a  failed  change  in  change  metrics. 

-  Aborted— the  change  failed,  accompanied  by  documentation  of  what  went  wrong. 

-  Completed  Successfully— the  change  was  implemented  and  is  functioning  appropriately. 

Items  not  to  do: 

•  Do  not  authorize  changes  without  rollback  plans  that  everybody  reviews.  Changes  do  fail,  so  be  proactive 
and  think  ahead  about  how  to  recover  from  a  problem  rather  than  attempting  to  do  so  during  the  heat  of 
firefighting. 

•  Do  not  allow  "rubber  stamping"  approval  of  changes. 

•  Do  not  let  any  system  changes  off  the  hook— someone  made  it,  so  understand  what  caused  it. 

•  Do  not  send  mixed  messages.  Bear  in  mind  that  the  first  time  the  process  is  circumvented,  incredible  damage 
can  be  done  to  the  process.  "Well  heck,  we  did  it  last  time"  or  "The  boss  said,  'just  do  it'"  both  send  the 
wrong  messages. 

Do  not  expect  to  be  doing  "closed  loop"  change  management  from  the  start.  Awareness  is  better  than  being 
oblivious,  and  managed  is  better  than  unmanaged.  Crawl,  walk,  run— when  you  put  in  a  valve,  you  put  it  in 
the  open  position  then  you  constrict  as  you  have  confidence  that  everything  is  flowing  through  it.  The  same 
is  true  for  change  management.  Start  with  a  particular  class  of  changes  and  constantly  refine  the  process. 
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The  Spectrum  Of  Change: 

The  management  of  change  is  an  evolutionary  process.  Groups  should  not  become  discouraged  as  they  start 
developing  their  change  management  processes.  The  solutions  may  require  changing  people,  processes,  and 
technology.  The  following  illustrates  the  stages  of  change  management: 

1.  Oblivious  to  Change— "Hey,  did  the  switch  just  reboot?" 

2.  Aware  of  Change— "Hey,  who  just  rebooted  the  switch?" 

3.  Announcing  Change— "Hey,  I'm  rebooting  the  switch.  Let  me  know  if  that  will  cause  a  problem." 

4.  Authorizing  Change— "Hey,  I  need  to  reboot  the  switch.  Who  needs  to  authorize  this?” 

5.  Scheduling  Change— "When  is  the  next  maintenance  window— I'd  like  to  reboot  the  switch  then?" 

6.  Verifying  Change— "Looking  at  the  fault  manager  logs,  I  can  see  that  the  switch  rebooted  as  scheduled." 

7.  Managing  Change— "Let's  schedule  the  switch  reboot  to  week  45  so  we  can  do  the  maintenance  upgrade 
and  reboot  at  the  same  time." 

What  You  Have  Built  And  What  You  Will  Likely  Hear 

In  this  phase,  our  goal  was  to  reduce  the  amount  of  time  we  are  spending  on  unplanned  work  down  to  25% 
or  less,  by  reducing  the  number  of  self-inflicted  problems  and  modifying  how  problems  are  solved  so  that 
change  is  ruled  out  early  in  the  repair  cycle.  By  increasing  the  change  success  rate  and  reducing  MTTR,  we  have 
not  only  decreased  the  amount  of  unplanned  work,  but  also  increased  the  number  of  changes  that  can  be 
successfully  implemented  by  the  organization. 

This  section  of  the  book  is  not  intended  to  be  a  complete  reference  document  on  change  management 
processes.  For  further  information,  please  refer  to  the  chapter  on  change  management  in  the  ITIL  Service 
Support  volume.  Copies  of  that  book  can  be  ordered  from  the  ITPI  Web  site. 

" The  first  step  is  hard,  but  not  nearly  as  hard  as  you  might  think,  and  the  rewards  are  worth  it.  You  are 
changing  people's  habits  and  ways  of  doing  work,  but  don't  forget  that  you  are  fundamentally  making 
peoples'  jobs  easier  as  well.  You  are  gaining  control  and  creating  a  stable,  maintained  and  predictable 
environment  for  people  to  work  in.  In  my  case,  it  took  a  security  event  to  initiate  this  process,  but  we  were 
able  to  remove  developer  access  to  production  systems,  remove  unnecessary  root  privileges,  and  start  down 
the  path  of  building  a  functional  change  process.  By  doing  the  steps  outlined  in  phase  one  of  Visible  Ops, 
we  were  able  to  reduce  the  catastrophic  impact  of  change,  and  we  bought  ourselves  90  days  during  which 
we  completed  the  rest  of  the  Visible  Ops  phases.  We  involved  the  entire  IT  crew,  who  were  very  responsive 
and  receptive,  as  were  the  other  decision  makers  and  stakeholders  involved."— joe  judge,  former 
INFORMATION  SECURITY  OFFICER  (ISO),  ADERO,  INC. 
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"The  value  of  the  change  management  processes  and  detective  controls  is  at  this  point  incalculable  to 
our  organization.  The  old  adage  that  "what  gets  inspected  gets  respected"  applies  here,  ,4s  a  result  of 
increasing  our  awareness  of  change  and  the  effects  of  change,  we  have  changed  the  culture  in  regards 
to  change.  We  are  a  more  thoughtful  organization;  more  conscious  of  change  and  more  judicious  in 
our  use  of  change.  The  Change  Advisory  Board  meetings  have  accomplished  far  more  than  the 
conscious  planning  of  specific  changes.  They  have  raised  the  consciousness  of  change  and  the  potential 
impact  of  change  across  the  board.  The  fact  that  we  are  managing  change  at  all  creates  this 
consciousness,  which  has  a  ripple  effect  throughout  the  organization. 

It  is  not  that  we  don't  make  mistakes  anymore;  but  we  have  become  more  scientific  in  our  approach 
to  mistakes;  mistakes  are  seen  more  as  learning  experiences  and  the  mistakes  have  become  fewer  and 
farther  between.  The  processes  and  detective  controls  have  helped  us  realize  many  of  our  goals  in  the 
pursuit  of  world-class  IT  management."— steve  darby,  vp  of  operations,  ip  services 

* The  first  phase  of  Visible  Ops  hits  the  nail  right  on  the  head  with  the  focus  on  Change  Management. 
In  any  IT  Service  Model  that  I  am  aware  of,  Change  Management  is  at  the  very  core.  Starting  with 
other  processes  (e.g.  Help  Desk/Incident  Management )  is  merely  staunching  the  bleeding,  without 
addressing  the  underlying  trauma.’’— jan  vromant,  itsm  consultant 

“We  have  seen  tremendous  value  after  implementing  a  change  management  process  and  detective 
controls.  Unnecessary  changes  were  identified  and  eliminated.  This  saved  valuable  time  and  moved  us 
to  a  more  proactive  environment.  Systems  upgrades  were  reduced  to  mere  hours  instead  of  days 
as  the  processes  were  streamlined  and  standardization  was  easier  to  attain.”— karen  fragale, 
data  center  manager 


The  benefits  generated  in  this  phase  are: 

•  We  have  increased  availability. 

•  We  have  reduced  the  time  spent  firefighting. 

•  We  have  increased  the  change  success  rate. 

•  We  have  created  a  formal  change  management  process  that  is  both  documented  and  adhered  to. 

•  We  have  reduced  the  risks  of  change  that  could  negatively  impact  production. 

•  We  have  made  failed  change  less  costly  and  more  visible  by  restricting  changes  to  planned  maintenance 
windows. 

•  We  have  reduced  MTTR  by  ensuring  that  causal  information  is  used  by  problem  management, 
pulled  from  our  change  management  processes. 

•  We  have  clearly  defined  the  IT  operations  and  security  roles,  welding  them  together  in  the 
change  and  problem  management  processes. 

•  We  have  improved  the  working  relationship  and  communication  between  the  functional  roles. 
They  are  now  working  together  to  solve  common  business  objectives,  reducing  the  number  of 
"drive-by"  surprises. 

•  We  have  started  creating  a  culture  of  change  management,  where  the  controls  are  owned  by 
operations  and  security. 
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Helpful  Tips  When  Preparing  for  an  Audit: 

•  Avoid  at  all  costs  creating  an  adversarial  relationship  with  auditors.  Instead,  demonstrate  that  you  have 
effective  management  and  control  processes  in  place,  and  the  documentation  to  prove  it.  If  you  cannot 
show  intended  and  actual  activities,  auditors  go  into  "archaeology"  mode.  (The  worst  thing  you  can  do 
is  become  defensive  and  adversarial,  especially  if  material  control  weaknesses  do  indeed  exist.) 

•  Make  sure  you  have  an  up-to-date  document  describing  your  change  management  process.  Show  this  to 
auditors  up  front  to  illustrate  what  you  want  to  be  measured  against.  Without  it,  they  will  bring  in  their 
own  processes  to  measure  you. 

•  Take  good  meeting  minutes  during  the  CAB  meetings  and  file  them.  Make  sure  they  are  dated.  Show 
meeting  minutes  to  auditors  to  demonstrate  that  the  meetings  are  actually  taking  place. 

•  The  mantra  of  post-Enron  auditors  is,  "If  it's  not  documented,  it  doesn't  exist."  Therefore,  be  sure  to 
document  both  your  work  and  your  meetings.  The  correct  level  of  documentation  should  be 
commensurate  with  the  level  of  risk  associated. 

•  To  show  that  your  change  management  processes  function,  meeting  minutes  should  show: 

-  Newly  authorized  and  scheduled  change  requests. 

-  Acceptance  of  implemented  changes  with  correlation  between  detected  changes  and  implemented 
changes,  showing  successful  implementation,  acceptance  by  a  change  manager  and  closure  of  the 
work  order. 

-  Changes  to  production  equipment  tracked  in  work  logs/work  order  tickets.  These  should  identify  the 
date,  time,  implementer  and  system  along  with  details  of  the  changes  made. 

•  Assemble  a  list  of  changes  made  outside  of  the  change  management  policy  and  corrective  actions  taken. 

•  On  a  regular  basis,  create  and  review  a  report  with  the  number  of  changes  requested,  changes  approved, 
MTTR  and  Change  Success  Rate  by  asset,  functional  area  and  organization,  etc. 

•  Engineer  the  change  workflow  and  ticketing  systems  in  such  a  way  that  "closing"  a  request  or  ticket  is 
not  possible  until  it  has  been  reviewed  and  accepted  by  the  change  manager.  This  ensures  accountability, 
visibility  and  fact-based  management,  instead  of  belief-based  or  faith-based  management. 

•  By  following  these  tips,  you  prove  that  you  have  functional  preventive,  detective  and  corrective  controls 
in  place.  For  more  information,  refer  to  Appendix  A. 
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Phase  Two:  "Catch  &  Release"  And  "Find  Fragile 
Artifacts"  Projects 

The  second  phase  of  Visible  Ops  focuses  on  creating  and  maintaining  an  inventory  of  production  assets.  Prior 
to  the  first  phase,  there  was  uncontrolled  change  and  many  different  configurations  in  production.  These 
configurations  must  be  inventoried  and  analyzed  to  determine  how  to  reduce  configuration  count,  which  will 
be  the  focus  in  Phase  Three. 


Issues  And  Indicators 

The  second  phase  of  Visible  Ops  tackles  the  following  issues: 


Issue 

Narrative  Example 

Inability  to  figure  out 
where  to  start  building  a 
configuration  management 
database  (CMDB)  and  a 
service  catalog  that  shows 
what  services  IT  provides. 

"Sometimes  walking  through  the  data  center  is  a  petrifying  experience.  There  are  hundreds 
of  servers,  which  all  look  alike,  that  all  seem  to  be  generating  firefighting  situations.  Which 
of  these  is  really  the  most  critical  to  our  business?  Is  there  a  better  way  to  find  all  the 
infrastructure  dependencies  than  seeing  them  catch  on  fire  when  we  make  a  change? 

"For  that  matter,  what  in  the  world  do  all  these  servers  actually  do?  Sometimes  when  one  of 
them  goes  belly-up,  we  get  screamed  at  by  someone  we've  never  heard  of.  Normally,  we 
wouldn't  even  listen  to  these  people  we  don't  know,  but  when  they  have  VP'  in  their  title, 
we  sometimes  can't  help  but  scramble  when  they  start  yelling.  How  can  we  establish  some 
prioritized  list  of  services  we're  providing  so  we  can  do  better  triage?" 

Inability  to  start  moving  from 
"individual  knowledge"  to 
"tribal  knowledge'.' This  is 
especially  challenging  when 
no  documented  processes 
exist  that  describe  what  the 

IT  operations  staff  is 
responsible  for. 

"After  weeks  of  cataloging  all  the  IT  services  that  we're  responsible  for,  we  finally  have  an  up- 
to-date  inventory.  In  fact,  we  also  now  have  an  asset  inventory  of  the  infrastructure  that 
each  service  depends  on.  This  still  doesn't  solve  the  problem  that  only  a  couple  of  rocket 
scientists  understand  how  to  run  some  of  these  assets. 

"For  instance,  take  that  DHCP  server  sitting  over  there.  We  know  that  if  we  turned  it  off,  it 
would  take  down  virtually  all  the  middleware  servers,  but  the  engineer  who  set  it  up  was  a 
college  intern  four  years  ago  who  left  after  his  internship  ended.  No  one  knows  how  to  fix 
this  thing.  Virtually  every  attempt  to  modify  this  box  results  in  a  catastrophic  failure  and 
man-weeks  of  work  trying  to  get  everything  up  and  running  again!  How  do  we  make  the 
DHCP  server  less  fragile?" 

Servers  become  like  snow¬ 
flakes:  Configuration  drift 
creeps  into  mission-critical 
infrastructure,  creating 
anomalous  personalities  in 
what  should  be  identical 
infrastructure. 

"Here's  an  interesting  question  for  you.  How  in  the  world  did  that  DHCP  server  even  get 
there?  Did  you  notice  that  the  DHCP  server  is  actually  running  on  one  of  our  four  DNS 
servers?  For  seven  years,  each  of  those  four  DNS  servers  were  exactly  the  same,  until  that 
college  intern  decided  to  commandeer  one  of  them  for  his  little  project. 

"Obviously,  when  we  first  found  that  rogue  DHCP  server,  we  tried  to  kill  it,  but  then 
found  out  during  the  middle  of  the  trading  day  that  all  the  middleware  servers  depended 
upon  it.  We  never  got  around  to  getting  rid  of  it,  and  the  configuration  variance  is 
definitely  starting  to  take  a  toll  on  other  operational  tasks.  For  instance,  applying  patches 
to  all  the  DNS  boxes  is  now  starting  to  have  radically  different  results  because  of  the 
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Implement  A  "Catch  &  Release"  Project 


"Insanity  is  doing  the  same  thing  over  and  over,  and  expecting  a  different  result."— attributed  to 
ALBERT  EINSTEIN 

By  completing  the  first  phase,  we  started  to  control  how  changes  are  made  in  production  to  reduce  the 
likelihood  of  risky  changes.  In  this  phase,  we  will  inventory  all  managed  assets,  and  then  identify  those  that 
create  the  most  unplanned  and  unscheduled  work.  When  we  complete  this  phase,  we  will  know  exactly  what 
infrastructure  needs  to  be  worked  on  by  the  release  engineering  team  in  the  next  phase. 

In  this  step,  we  will  do  exactly  what  the  park  rangers  for  the  National  Wildlife  Service  do,  which  is  "Catch  & 
Release."  Their  job  is  to  bag  and  tag  each  animal  in  the  national  parks,  picking  them  up,  weighing  them, 
counting  how  many  legs  they  have,  giving  them  names,  finding  out  if  they  already  have  a  record,  and  so  forth. 

Our  goal  in  this  step  is  to  capture  all  equipment  in  the  data  center.  For  each  asset  captured,  figure  out  what  it's 
running,  what  services  depend  upon  it,  who  has  primary  management  responsibility  for  it,  how  fragile  this 
infrastructure  is,  and  so  forth.  Attempting  to  find  out  this  information  in  the  heat  of  firefighting  contributes 
to  miscommunication  and  often  results  in  tremendous  stress  and  frantic  phone  calls.  We  will  perform  this 
inventory  when  infrastructure  is  not  on  fire,  so  that  the  information  will  be  readily  available  in  case  of 
emergency.  Note  that  the  inventory  is  not  only  for  use  in  problem  management  scenarios,  but  to  guide 
resource  deployment  in  Phases  Three  and  Four,  too. 

While  analyzing  assets,  ask  these  important  questions: 


•  What  does  it  do? 

•  What  is  the  hardware  platform? 

•  What  is  the  operating  system  platform? 

•  What  applications  are  installed? 

•  Who  is  responsible  for  this  asset's  uptime? 

•  What  service(s)  does  it  support? 

•  Who  is  authorized  to  make  changes? 

•  What  does  this  box  do  for  the  business? 

•  What  will  happen  when  this  box  stops  working 
completely? 

•  What  will  happen  when  the  performance  of  this  box  is 
severely  degraded? 

•  What  is  the  change  success  rate? 

•  Is  this  device  fragile?  Can  we  build  a  new  one  if  it  fails? 
(See  the  next  Visible  Ops  Phase  3:  Create  a  Repeatable 
Build  Library.) 

•  What  are  its  dependencies? 

•  What  other  infrastructure  depend  on  this  unit? 

•  What  planned  and  unplanned  changes  have 
been  made? 

•  What  is  the  device's  name?  Is  it  appropriate  for  the 
tasks  performed? 


•  What  is  the  outage  cost?  (In  other  words,  the  cost  per 
minute  of  downtime) 

•  Where  is  it  physically  located? 

•  Is  there  anything  odd  about  this  box? 

•  Is  this  a  generally-supported  platform  in  our  company? 

•  Is  this  box  going  to  go  away  in  the  next  few  months? 

•  How  do  we  get  access  to  this  box  (remote  or  otherwise)? 

•  How  is  this  unit  backed  up? 

•  How  long  does  it  take  to  re-provision  this  unit 
(estimate)? 

•  How  long  can  the  business  afford  to  be  without  it? 

•  Are  we  monitoring  this  unit  for  changes  (i.e.  is  a 
detective  control  installed)? 

•  Are  we  fault-monitoring  this  device? 

•  Do  the  fault-monitoring  assumptions  match  the 
dependency  realities? 

•  If  the  unit  is  mission  critical,  then  are  there  adequate 
hardware  backups  in  place  (power  supply,  network  card, 
RAM,  etc)? 

•  Why  do  we  feel  that  this  unit  is  unstable  (if  applicable)? 

•  Is  there  anything  that  needs  attention  on  this  unit? 
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Because  collecting  this  information  may  seem  like  tedious  data  entry,  you  may  be  tempted  to  assign  junior 
staff  to  this  project.  Do  not  fall  into  this  trap!  The  goal  of  this  step  is  to  capture  the  information  only  known 
by  the  most  senior  staff,  and  use  the  information  to  create  repeatable  and  verifiable  processes.  The  senior  staff, 
due  to  their  experience  and  accumulated  knowledge,  will  provide  the  bulk  of  this  information  that  has  not 
been  previously  captured.  Currently,  problems  may  escalate  to  senior  staff  simply  because  no  documentation 
exists  to  allow  more  junior  staff  to  resolve  them. 

Find  Fragile  Artifacts 

" When  you're  in  an  earthquake  on  a  unicycle,  juggling  chain  saws,  the  only  way  to  survive  is  to  tack  down 
everything  you  can  tack  down,  so  you  can  deal  with  what  you  can’t.”— Stephen  chakwin 

Most  data  centers  usually  contain  numerous  infrastructure  that  are  considered  "dangerously  fragile."  During 
the  "Catch  and  Release"  process,  these  fragile  artifacts  must  be  tagged  and  treated  as  such.  The  senior  staff 
usually  knows  where  the  fragile  artifacts  are  because  they  instinctively  shy  away  from  them— knowing  that  if 
someone  even  looks  at  one  wrong,  it  will  crash  and  cause  a  massive  episode  of  unplanned  work.  Infrastructure 
is  fragile  when  it  has  a  low  change  success  rate  and  a  high  MTTR.  In  other  words,  fragility  occurs  when  all  changes 
are  risky  and  potentially  require  a  rebuild  from  scratch,  resulting  in  several  man-weeks  of  work  to  repair. 

Once  we  have  a  list  of  fragile  infrastructure,  what  do  we  do  with  it?  First  and  foremost,  avoid  making  any 
changes  to  them!  Consider  how  valuable  it  was  to  avert  risky  changes  during  the  change  management 
meetings.  By  flagging  fragile  infrastructure  that  generates  inordinate  amounts  of  unplanned  work,  we  can 
further  avoid  risky  changes.  During  this  step,  some  groups  have  literally  put  Post-It  notes  on  infrastructure 
boldly  warning  "Do  Not  Touch!"9  Others  use  login  banner  screens  or  announcements  to  convey  the  same 
message.  And  of  course,  these  are  the  systems  that  need  detective  controls  in  place,  to  ensure  that  the  change 
management  process  is  not  circumvented! 

Beyond  the  value  of  deterring  changes,  the  "Do  Not  Touch”  sign  has  further  value  because  it  can  allow  us  to  make 
better  decisions  in  the  CAB  meetings.  What  is  the  value  of  preventing  a  change  that  would  have  resulted  in  200 
man-hours  of  unplanned  and  unscheduled  work?  There  is  a  well-documented  story  of  a  $239  million  weather 
satellite  that  was  accidentally  dropped  because  someone  had  removed  the  bolts  that  secured  it  on  the  stand,  and 
did  not  bother  to  put  a  sign  up  warning  people  not  to  touch  the  delicate  satellite!10 

Throughout  all  four  phases  of  Visible  Ops,  we  foster  and  positively  reinforce  organizational  learning.  Einstein 
once  defined  insanity  as  "doing  the  same  thing  over  and  over,  and  expecting  different  results."  Here,  we 
document  what  works  and  what  does  not  work  to  guide  decision  making,  and  to  avoid  actions  that  historically 
generated  unplanned  and  unscheduled  work.  We  will  leverage  these  findings  in  the  third  phase  to  increase  the 
number  of  planned,  tested,  stable,  and  repeatable  configurations. 

Prevent  Further  Configuration  Mutation 

"Do  not  look  where  you  fell,  but  where  you  slipped."— African  proverb 

Uncontrolled  changes  on  systems  cause  them  to  deviate  from  any  known  and  trusted  states.  We  may  deploy  one 
thousand  servers  that  are  initially  identical,  but  when  their  configurations  drift  over  time,  they  become  like 


9  Bill  Shinn  tells  the  story  of  a  former  Dutch  boss  who  would  walk  around  the  server  room  placing  "Do  Never  Touch”  Post-It  notes  on 
the  critical  servers. 

“Fordahl,  Matthew.  "Under-construction  satellite  topples  to  floor  in  mishap."  The  Associated  Press,  9/10/2003. 
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snowflakes— each  different  and  impossible  to  reproduce.  It  is  absolutely  critical  that  while  we  inventory 
systems  in  this  entire  "Catch  &  Release"  phase,  they  must  not  mutate  into  some  other  unrecognizable 
configuration.  (Recall  the  story  of  the  DNS  server  suddenly  becoming  a  DHCP  server  as  well.) 

During  Inventory  Projects: 

While  we  are  doing  the  "Catch  &  Release"  inventory,  we  must  freeze  configurations.  If  new  builds  are  silently 
deployed  into  production  or  previously  inventoried  builds  change,  we  jeopardize  the  accuracy  of  our  inventory. 
Therefore,  we  must  do  the  following: 

1.  Create  a  clear  mandate  delaying  deployment  of  new  builds  and  changes  to  existing  builds  until  the 
inventory  process  is  over.  Set  clear  goals  for  dealing  with  each  fragile  patient,  defining  the  start  and  finish 
of  the  change-free  moratorium,  and  communicating  with  business  units  to  define  a  mutually  acceptable 
change  freeze  window.  The  last  thing  we  want  to  do  is  unnecessarily  delay  change  and  prevent  a  new 
product  rollout— that's  as  bad  as  an  unplanned  outage  from  the  perspective  of  a  business  unit. 

2.  The  "Catch  and  Release"  and  "Find  Fragile  Artifacts"  steps  must  happen  quickly.  It  is  unrealistic  to  prohibit 
changes  for  too  long.  This  may  force  people  to  circumvent  the  change  process  in  order  to  get  critical  work 
done. 

3.  Allow  truly  urgent  and  necessary  changes  to  be  made  through  the  CAB/EC  meetings,  making  sure  to 
capture  any  changes  to  systems  that  are  only  partially  completed  with  their  Catch  and  Release  treatment. 

During  Ongoing  Operations  After  Completion  Of  "Catch  and  Release": 

An  automated  detective  control  must  be  used  as  a  ratchet  to  ensure  that  any  progress  made  is  not 
immediately  lost  due  to  entropy.  This  control  must  detect  all  changes,  notify  us  when  processes  have  been 
circumvented,  and  enable  an  update  to  the  configuration  inventory.  Detected  changes  that  cannot  be 
mapped  to  authorized  work  must  be  rolled  back. 

What  You  Have  Built  And  What  You  Will  Likely  Hear 

“Every  time  a  security  or  IT  organization  has  done  an  asset/service  inventory  I've  seen  only  two  types  of 
reactions.  On  the  mild  side,  you  have  'That’s  odd.  Where'd  those  come  from?'  On  the  stronger  side, 
which  is  more  typical,  you  have  ‘Houston,  we  have  a  problem!' 

Seek  and  ye  shall  find— what  you  have  never  matches  what  you  think  you  have.  The  best  organizations 
merely  keep  that  gap  small,  safe  and  manageable."— joe  judge,  former  iso,  adero,  inc. 

“We  found  that  creating  and  managing  asset  and  service  inventories  is  essential  to  achieving  change 
management.  Our  asset  inventory,  in  the  form  of  a  Configuration  Management  Database,  gives  us  a 
reference  point  as  to  what  the  configuration  of  the  IT  infrastructure  should  be  as  well  as  a  record  of 
changes  to  the  infrastructure.  The  CMDB  supplies  a  contextual  environment  for  managing  change  that 
enables  management  of  change  from  the  perspectives  of  relationships  between  entities  as  well  as  history. 

The  CMDB  quickly  became  an  essential  reference  for  change  management.  It  is  hard  to  imagine 
managing  change  without  an  accurate  CMDB. 
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In  a  way,  the  service  inventory  fills  a  role  that  is  similar  to  the  asset  inventory.  It  is  another  reference,  this 
time  describing  the  policies  and  procedures  that  govern  operational  activities  that  result  in  changes  being 
made  to  the  configuration  of  assets.  Solid,  detailed  services  documentation  is  another  essential  element  for 
successful  service  management.  By  documenting  the  services  in  detail,  including  how  the  services  are  to  be 
implemented,  we  have  improved  our  control  over  service  management.  Failed  changes  can  usually  be  traced 
back  to  someone  not  following  documented  procedure  or  an  incorrect  or  incomplete  description  of  a 
procedure.  By  documenting  known  good  procedures  and  ensuring  that  staff  is  trained  to  follow  the 
procedures,  we  have  a  powerful  system  for  increasing  our  change  success  rate. 

Services  and  assets  can  be  seen  as  the  inputs  and  outputs  of  the  change  management  process.  By 
documenting  the  types  of  services  that  are  authorized  to  be  performed  and  documenting  the  status  of  the  IT 
infrastructure  upon  which  they  are  performed  we  have  completed  the  system  that  is  required  to  track  change 
scientifically  and  generate  management  feedback  in  a  context  that  enables  continuous  process 
improvement."— steve  darby,  vp  of  operations,  ip  services 

"The  identification  of  the  most  fragile  items  in  the  IT  architecture  is,  in  my  humble  opinion,  a  crucial  second 
step,  and  I  was  excited  by  the  fact  that  Visible  Ops  recognizes  it  as  such.  I  once  heard  a  well-known 
consultant  say  'configuration  management  has  to  be  done  on  the  sly,  as  you  can  not  get  any  wins  from 
there.'  Nothing  could  be  further  from  the  truth,  and  the  second  phase  of  Visible  Ops  proves  it  in  a  logical 
way!  Configuration  management  and  change  management  are  two  sides  of  the  same  coin,  and  one  cannot 
work  without  the  other.  Phase  Two  of  Visible  Ops  focuses  on  the  core  configuration  management— right  on 
target!"— jan  vromant,  itsm  consultant 

"It  has  always  been  important  to  keep  an  asset  inventory.  It  is  also  extremely  important  to  inventory  your 
services  and  processes.  It  has  helped  move  our  organization  from  a  reactive  environment  to  a  proactive 
environment."— karen  fragale,  data  center  manager 


The  benefits  generated  in  this  phase  are: 

•  We  have  created  a  service  catalog  that  documents  the  most  critical  services  we  are  supporting. 

•  We  have  documented  what  those  services  are  and  what  infrastructure  supports  them. 

•  We  have  created  a  configuration  management  database  (CMDB)  that  illustrates  the  mapping  between  services 
and  infrastructure,  showing  the  relationships  between  all  the  configuration  items  (Cl).  Each  Cl  is  associated 
with  a  service  and  may  have  other  CIs  associated  with  it.  An  example  CMDB  table  structure  is  provided  in 
Appendix  E. 

•  We  have  created  decision  support  tools  and  metrics  that  increase  change  success  rates  and  further  decrease 
unplanned  work. 

•  We  have  fostered  organizational  learning  by  using  historical  change  success  information  to  make  better  risk- 
driven  decisions  around  change. 

•  We  have  created  a  prioritized  list  of  projects  that  our  release  engineering  team  will  work  on,  to  replace  fragile 
artifacts  with  stable  infrastructure. 

•  We  have  implemented  detective  controls  to  ensure  our  inventory  information  is  always  up  to  date.  The  detec¬ 
tive  controls  now  are  being  used  for  both  problem  management  and  ensuring  the  integrity  of  the  CMDB. 
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Helpful  Tips:  When  Preparing  for  an  Audit 

•  Be  able  to  show  the  process  of  how  you  generated  the  catalog  of  services  and  the  assets  that  support  them. 
Remember,  the  inventory  should  include  both  hardware  and  software. 

•  Show  that  you  understand  the  business  processes  you  are  supporting  by  working  with  senior  management 
to  rank  the  services  by  importance  to  the  organization  and  their  degree  of  fragility. 

•  Show  how  you  assure  that  the  inventory  is  maintained  and  accurate.  The  lack  of  an  accurate  inventory 
may  indicate  to  auditors  that  there  are  inadequate  controls. 

•  Show  the  list  of  fragile  artifacts  resulting  from  this  phase  as  evidence  that  you  are  performing  risk  mitigation. 

•  Be  able  to  document  the  systems  and  processes  used  to  detect  changes. 
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Phase  Three:  Create  A  Repeatable  Build  Library 

In  phase  two,  we  inventoried  all  of  the  IT  services  and  the  dependent  infrastructure.  We  identified  fragile 
artifacts  with  low  change  success  rates  and  high  MTTR  because  of  their  contribution  to  unplanned  work.  Then 
we  plastered  them  with  Post-It  notes  warning  "Do  Not  Touch!" 

In  this  phase,  we  will  create  a  library  of  repeatable  builds  focusing  first  on  fragile  infrastructure.  To  accomplish 
this,  we  must  define  build  mechanisms,  create  system  images,  and  establish  documentation.  What  results  is  a 
repeatable  process  for  building  infrastructure  from  "bare  metal." 

By  making  infrastructure  easier  to  rebuild  than  repair,  we  will  create  the  data  center  equivalents  of  fuses:  when 
things  go  wrong  with  production  fuses,  we  do  not  repair  them;  instead,  we  pop  them  out  and  re-provision 
them  from  scratch.  However,  success  in  this  phase  hinges  on  our  ability  to  control  production  changes  to  that 
fuse,  lest  the  new  fuse  have  radically  different  behavior  than  the  fuse  it  replaces.  This  technique  replicates 
what  all  the  high-performing  organizations  do,  resulting  in  the  high  server/admin  ratios,  low  amounts  of 
unplanned  work,  and  the  ability  to  maintain  manageable  configuration  counts. 

Issues  And  Indicators 


1  Issues 

Narrative  Example 

Critical  production  infra¬ 
structure  resembles  works 
of  art,  which  can  never  be 
replaced,  even  during 
disaster  recovery  situations. 

"Remember  that  supposedly  simple  'DHCP  server'  1  was  talking  about?  Let  me  tell  you  about 
our  last  attempt  to  replace  it.  In  fact,  1  have  an  email  from  the  build  engineer  that  wrote  me 
during  our  last  disaster  recovery  test  last  spring.  It  reads: 

Dear,  Bob.  1  am  writing  this  to  you  from  the  data  center  that  supports  the  billing  middleware.  1  do  not 
need  to  remind  you  that  we  haven 't  turned  this  stuff  off  in  over  four  years.  We  are  going  to  be  power- 
cycling  it  in  ten  minutes.  No  one  on  the  team,  myself  included,  knows  whether  we  will  survive  the 
reboot.  Worse  yet,  if  it  doesn't  fully  boot  up  on  its  own,  we  will  have  no  idea  what  we  need  to  do  to 
bring  it  back  up.  That's  why  the  entire  staff  is  ready  to  spend  the  weekend  on  site. 

Wish  me  luck. 

Ellen 

"Well,  72  hours  later,  we  got  it  up,  but  just  barely.  If  Ellen  hadn't  been  able  to  hunt  down 
people  who  managed  this  stuff  four  years  ago,  1  don't  think  we  could  have  pulled  it  off.  Talk 
about  archaeology — what  would  have  happened  if  one  of  those  people  were  no  longer 
alive?  What  if  we  didn't  have  access  to  Google?  How  in  the  world  can  1  make  sure  this 
never  happens  to  me  again?" 

Configuration  drift  under¬ 
mines  release  management 
investments.  In  the  absence  of 
"refrigeration,"  infrastructure 
"spoils"  in  production. 

"So  what  is  the  point  of  having  our  release  engineers  work  on  repeatable  builds  for  the  DNS 
servers,  when  the  instant  they  go  into  production,  they  get  patched,  changed,  configured, 
re-configured,  over  and  over  again.  Meanwhile,  the  release  engineers  are  working  on  their 
next  release,  spending  weeks  getting  it  loaded  up  into  Microsoft  SMS  or  Marimba,  blasting 
it  out  and  then  overwriting  all  the  production  changes  we  made!  Suddenly,  we  in  IT 
operations  have  to  reapply  all  the  changes  we  made  in  the  last  two  months.  Heaven  help 
us  if  we  didn't  document  all  of  our  changes. 

"1  probably  mentioned  that  we  sometimes  want  to  strangle  all  the  developers  because  of  all 
the  junk  they  throw  over  the  wall  at  us.  Unfortunately,  equally  often  these  days,  we  want  to 
strangle  the  release  engineers  as  well." 
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Issues 

Narrative  Example 

Inability  to  decide  where  to 
apply  release  engineering 
resources:  what  builds  should 
they  be  working  on,  and 
when  is  it  good  enough  to 
move  on? 

"Because  of  all  the  configuration  variance,  the  release  engineers  have  a  difficult  time 
determining  how  to  reduce  configuration  complexity.  Even  worse,  the  production 
environment  is  a  moving  target,  because  it  changes  so  quickly. 

"Although  release  engineering  tries  to  keep  up  with  the  production  environment,  they  rarely 
can,  due  to  all  of  our  patches  and  modifications.  Inevitably,  we  have  to  spend  a  lot  of  time 
rebuilding  their  boxes  and  doing  testing  until  we  hope  we've  caught  all  the  possible 
errors.  As  you  can  imagine,  we  definitely  miss  problems  once  in  a  while  and  they  pop  up 
during  the  worst  times!" 

Downward  spiral  of  increasing 
configuration  counts,  and 
increased  level  of  knowledge 
necessary  to  resolve  problems, 
which  increases  the  need  for 
more  highly  skilled  system 
administrators. 

"Quite  frankly,  I'm  starting  to  really  understand  that  1  cannot  avoid  being  in  a  downward 
spiral.  My  production  configurations  continue  to  turn  into  irreplaceable  snowflakes,  which 
require  an  increasing  level  of  skill  to  make  any  changes  and  administer.  In  turn,  this  is 
increasing  my  need  for  rocket  scientists,  forcing  me  to  hire  ever  smarter  people  to  do  more 
non-repeatable  tasks.  If  1  keep  this  up,  then  I'll  have  one  rocket  scientist  for  each  server, 
who'll  be  sitting  on  the  bench  doing  nothing  most  of  the  time! 

"In  reality,  1  need  to  be  going  the  other  way.  1  need  to  get  my  rocket  scientists  out  of 
the  front  lines  and  building  tools  to  empower  the  production  staff  to  diagnose  and  fix 
the  problems!" 

Development  routinely  makes 
fixes  on  production  infra¬ 
structure  instead  of  providing 
tools  for  Ops  staff.  They 
"throw  the  pig  over  the  wall" 
to  Ops  with  little  ability  to 
control,  stage,  or  reject. 

"Humbly  speaking,  1  feel  that  1  have  enough  problems  with  the  rocket  scientists  in  the 
development  organization.  1  should  be  creating  a  process  where  application  developers  are 
better  integrated  with  operations.  Somebody  needs  to  take  the  time  to  package  them  for 
distribution  and  installation  in  a  way  that  doesn't  require  hundreds  of  hours  of  integration 
engineering,  and  that  can  be  accomplished  by  production  staff! 

"For  this  to  work,  the  production  staff  needs  to  have  the  ability  to  say  'no'  on  production 
changes.  If  pushing  out  a  release  has  a  25%  failure  rate,  then  we  need  to  be  able  to  push  it 
back  to  development  to  rework.  What  use  is  deploying  software  if  it  will  cause  huge 
amounts  of  unplanned  work  and  takes  down  revenue-generating  infrastructure?" 

Ops  routinely  crashes 
infrastructure  by  applying 
security  patches. 

"1  sat  in  a  SANS  class  on  applying  the  'Cold  Standard'  configuration  to  Windows  servers. 
Definitely  all  smart  things  to  do,  but  do  you  want  to  talk  about  risk?  There  were  1 30  people 
taking  the  class,  each  person  bringing  their  own  laptop  to  learn  how  to  apply  the  changes. 

By  the  end  of  the  six  hours,  five  people  had  laptops  that  could  no  longer  boot! 

"It  finally  dawned  on  me.  Security  is  often  recommending  'fixes'  that  have  a  3-5%  failure  rate, 
which  is  the  reason  why  my  heart  stops  whenever  1  see  one  of  these  guys  submit  an 
'emergency  change  request'  to  apply  these  security  patches!" 

Ops  applies  infrastructure 
patches  that  never  get 
reflected  in  future  builds. 

"This  makes  it  all  the  more  important  that  if  we  go  through  the  agony  of  figuring  out  how 
to  successfully  deploy  these  security  patches,  they  absolutely  must  make  it  into  the  stored 
golden  builds.  Otherwise,  we're  going  to  have  to  jeopardize  the  infrastructure  and  spend 
hundreds  of  man-hours  redoing  all  the  work!" 

The  production  team  supports 
unusual  applications  and 
infrastructure. 

"How  in  the  world  did  we  get  into  the  position  where  we  are  supporting  fourteen  different 
database  applications?  I'm  not  even  sure  1  know  what  a  'Kumquat  Express  Postgres  IV'  is, 
but  it  catches  on  fire  a  lot,  we  know  how  to  reboot  it,  but  that's  about  it.  1  think  it  was 
some  marketing  person  who  bought  it  on  their  credit  card,  put  a  pig  around  it,  and  threw 
it  over  the  wall  at  us." 

Create  A  Release  Management  Team 

In  this  phase,  our  goal  is  to  take  the  most  senior  IT  operations  staff  and  move  them  from  a  reactive  firefighting 
role  to  a  proactive  release  management  function,  where  they  are  constantly  working  on  software  and 
integration  releases  that  will  get  deployed  into  production.  By  doing  this,  they  operate  early  in  the  IT 
operations  lifecycle  where  the  defect  costs  are  lowest.  Most  likely,  the  team  was  also  involved  with  the  previous 
"Catch  &  Release"  phase  and  is  very  aware  of  the  systems  that  are  in  production. 
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The  release  management  team’s  primary  role  is  building  the  mechanisms  to  deploy  the  best  configurations  into 
production.  They  do  not  do  the  builds— they  engineer  the  builds.  In  other  words,  they  design  the  builds,  but 
don't  build  the  servers.  In  this  phase,  their  first  task  is  to  create  repeatable  builds  for  the  most  critical  and 
fragile  infrastructure  assets  identified  during  the  "Catch  &  Release"  phase.  Their  objective  is  to  make  it  cheaper 
to  rebuild  infrastructure  than  to  repair.  Ideally,  this  will  be  done  by  taking  the  affected  infrastructure,  burning 
it  down  to  bare  metal,  and  rebuilding  it  by  pushing  a  button. 

In  increasing  order  of  importance,  the  following  typical  benefits  result  from  rebuilding  instead  of  repairing: 

•  Rebuilding  infrastructure  is  an  automated  process  and  takes  a  known  amount  of  time,  as  opposed  to 
firefighting  and  repairing,  which  almost  always  take  longer  than  we  originally  estimate. 

•  Rebuilding  infrastructure  tends  to  introduce  less  configuration  variance,  as  opposed  to  repeated  break/fix 
cycles  which  allows  additional  configuration  variance  to  creep  in. 

•  Because  the  rebuild  process  is  automated,  documented  and  less  complicated,  it  can  be  done  by  junior  staff, 
freeing  up  senior  staff  from  firefighting. 

•  When  senior  staff  become  free  from  firefighting,  they  can  keep  working  on  new  build  projects,  which  will  fix 
other  systemic  issues. 

The  release  management  team  continually  works  on  proactive  projects  that  reduce  the  likelihood  of  unplanned 
work,  systematically  eliminating  the  sources  of  disasters  before  they  strike.  They  strive  to  reduce  the  number  of 
unique  configurations  in  production  and  increase  the  lifespan  of  a  configuration  before  it  needs  to  be  changed  or 
replaced.  Both  of  these  goals  reduce  complexity  and  cost  while  improving  manageability. 
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Figure  5:  Staff  deployment  gradually  shifts  from  reactive  resolution  processes  to  proactive  release  processes 

By  moving  the  most  senior  staff  to  the  release  engineering  processes,  we  better  equip  the  more  junior  staff  to 
maintain  production  infrastructure.  By  allocating  staff  this  way,  the  organization  uses  people  more  effectively 
at  all  levels:  Their  mastery  of  configurations  continually  increases  while  they  integrate  it  into  documented  and 
repeatable  processes.  We  jokingly  refer  to  this  phenomenon  as  "turning  firefighters  into  curators,"  because  of 
the  tendency  for  the  best  and  the  brightest  engineers  to  be  the  worst  instigators  of  self-motivated  firefighting. 
By  putting  them  into  the  build  process,  they  become  protective  of  post-deployment  build  integrity,  and 
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actively  discourage  production  firefighting!  At  the  same  time,  more  junior  levels  of  the  organization  are  able 
to  take  over  problem  resolution. 

In  the  following  steps,  we  will  create  a  process  for  the  release  management  team  to  generate  repeatable  builds, 
store  them  in  the  definitive  software  library  (DSL),  and  put  together  maintenance  and  update  plans  for  the 
builds.  We  will  also  create  a  process  for  the  operations  team  to  take  the  builds  from  the  DSL,  and  then 
provision  them  into  production. 

We  will  describe  the  DSL  in  more  detail  in  the  next  several  steps. 

Create  A  Repeatable  Build  Process 

Now  that  we  have  created  a  release  management  team,  they  need  some  projects  to  work  on.  Not 
coincidentally,  this  is  the  output  of  the  "Find  Fragile  Artifacts"  step  of  the  previous  phase.  We  have  already 
created  a  prioritized  list  of  fragile  infrastructure  that  needs  to  be  replaced  with  stable  infrastructure  that  can 
be  repeatedly  rebuilt  by  junior  staff. 

Golden  builds  are  the  output  of  the  repeatable  build  processes.  They  are  considered  "golden"  because  they 
have  been  through  the  planning,  testing  and  approval  processes  prior  to  being  pushed  into  production.  Builds 
must  be  updated  when  new  patches  and  upgrades  are  integrated  and  new  applications  added. 

The  golden  builds  are  stored  in  the  DSL.  Think  of  the  DSL  as  the  vault  where  all  required  software  assets  reside. 
The  contents  of  the  DSL  are  used  to  construct  and  reconstruct  production  infrastructure.  It  includes  software 
media,  license  keys,  software  patches,  and  so  forth.  The  DSL  is  the  authoritative  and  secure  storehouse  of  all 
software  that  has  been  reviewed  and  approved  for  production  use. 
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Figure  6:  ITIL  "Definitive  Software  Library"  Diagram 
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To  define  which  build  projects  the  release  management  team  should  be  working  on,  do  the  following: 

1.  Identify  the  common  set  of  essential  services  and  components  used  across  your  infrastructure.  These  are  the 
lowest  common  denominators  that  apply  everywhere.  To  start,  consider  creating  lists  of  supported 
components  for  the  following  categories: 

a.  Infrastructure  and  operating  systems 

b.  Applications 

c.  Business  rules 

d.  Data 

2.  From  these  components,  create  a  list  of  standardized  components,  called  a  "build  catalog."  Look  for  ways 
to  create  components  that  can  be  reused  and  combined  to  create  standardized  configurations.  For  example, 
a  Web  server  build  may  be  composed  of  a  Solaris  build,  with  the  Apache  build  installed  over  it.  The  database 
server  build  may  be  the  same  Solaris  build,  but  with  the  Oracle  package  installed  over  it. 

3.  For  each  component  in  the  build  catalog,  create  a  repeatable  build  process  that  generates  it.  The  goal  is  to 
have  an  automated  build  system  that  can  provision  the  package  by  "pushing  a  button."  Examples:  For  AIX, 
use  NIM;  for  Solaris,  use  Jumpstart;  for  Windows,  use  InstallShield  AdminStudio  to  "diff"  systems  and  create 
HAL-proof  installers. 

4.  Any  testing  or  lab  environment  should  be  isolated  from  the  production  network  to  ensure  that  it  does  not 
disrupt  production  systems  and  to  make  sure  that  all  dependencies  outside  the  test  environment  are  fully 
documented  and  understood. 

5.  Ensure  that  you  can  recreate  each  system  with  a  bare  metal  build.  Our  goal  is  a  repeatable  process  that 
eliminates  anything  tedious  and  error-prone,  as  well  as  reducing  labor,  errors,  and  the  amount  of  spiritual 
energy  required  to  maintain  "snowflake  infrastructure."  (Note  that  VMware  is  a  useful  tool  to  create 
identical  virtual  servers  for  integration  testing.) 

6.  For  critical  high-availability  or  load-balanced  environments  where  many  machines  perform  the  exact  same 
function,  develop  reference  builds  that  can  provision  a  box  from  bare  metal  without  human  intervention. 

7.  When  the  build  engineering  process  has  been  completed,  store  them  in  the  DSL,  making  the  build  available 
to  the  provisioning  teams.  Creating  and  maintaining  the  DSL  is  covered  in  the  next  step. 

Create  And  Maintain  The  Definitive  Software  Library  (DSL) 

Ideally,  the  release  management  team  will  not  be  doing  the  actual  provisioning  and  deployment  of 
infrastructure  into  production.  Instead,  the  projects  they  complete  will  generate  builds  that  are  put  into  the 
DSL.  These  are  then  used  by  the  operations  team  to  deploy  the  various  builds  into  production. 

In  this  step,  we  will  describe  the  processes  for  how  we  will  create,  populate  and  maintain  the  DSL: 

Generate  The  DSL  Approval  Process 

1.  Designate  a  manager  to  maintain  the  DSL,  who  will  be  responsible  for  authorizing  the  acceptance  of  new 
applications  and  packages.  Of  course,  in  the  beginning,  the  DSL  will  be  empty,  which  brings  us  to  the  next  step. 
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2.  Create  an  approval  process  for  accepting  items  into  the  DSL.  For  example,  we  want  to  prevent  some 
business  manager  demanding  the  use  of  a  new  IIS  6.0  version  to  support  some  new  marketing  Web  site, 
which  then  requires  a  whole  new  production  skill  set  to  maintain.  Ideally,  acceptance  should  require 
approval  from  both  the  operations  and  release  management  staff,  as  well  as  a  relevant  expert.  Furthermore, 
the  prospective  builds  should  pass  a  pilot  or  lab  trial  prior  to  inclusion  into  the  DSL. 

3.  Establish  a  provisioning  vault  or  clean  room  to  store  approved  media  for  the  DSL  and  to  build  servers.  Its 
network  should  be  isolated  and  not  directly  connected  to  the  Internet  or  other  IT  networks. 

4.  Any  software  accepted  into  the  DSL  (both  retail  and  internally  developed  applications)  must  be  under 
revision  control. 

5.  Audit  the  DSL  to  ensure  that  it  contains  only  authorized  components. 

Place  Repeatable  Builds  Into  The  DSL 

1.  Initially,  put  all  the  running  applications  into  the  DSL,  under  a  special  "amnesty  program."  Because  we  do 
not  have  repeatable  builds  for  them  yet,  tag  these  as  having  amnesty  for  a  fixed  period,  e.g.  one  year. 

2.  For  each  application  in  the  amnesty  program,  create  a  repeatable  build  package,  using  as  many  of  the  generated 
builds  that  were  previously  defined.  Our  goal  is  to  replace  each  program  under  amnesty  with  as  many  pre¬ 
defined  components  as  possible,  to  best  leverage  economies  of  scale.  Benefits  include  the  ability  to  do  mass 
upgrades,  reduced  configuration  variance,  and  so  on.  For  example,  we  may  initially  have  14  different  Solaris 
and  Windows  builds,  but  our  goal  will  be  to  engineer3  builds  to  replace  them  all. 

3.  Periodically  review  the  DSL  to  weed  out  packages  that  are  no  longer  useful,  maintainable,  or  cost  effective 
to  keep  around.  For  example,  after  reviewing  requirements,  we  find  that  instead  of  five  database  packages, 
the  same  jobs  can  be  handled  by  just  three. 

Create  An  Acceptance  Process  Contract 

Often,  unclear  roles  for  the  release  management  and  production  teams  create  a  situation  where  they  undermine 
each  other,  instead  of  working  together  to  solve  common  objectives.  To  avoid  this,  create  clear  organizational 
responsibilities  and  shared  goals  for  both  teams,  and  define  the  interface  and  working  relationship  between 
them.  The  primary  mechanism  to  accomplish  this  is  the  acceptance  process,  where  the  production  organization 
decides  whether  or  not  to  accept  what  the  release  management  team  has  built. 

A  successful  acceptance  process  allows  both  teams  to  solve  the  common  business  problems,  despite  their 
slightly  different  objectives.  The  production  team  should  focus  on  their  desire  for  platform  stability,  ease  of 
manageability  for  production  staff,  predictability,  and  fully  tested  functional  products  (as  opposed  to  "do-it- 
yourself"  kits).  On  the  other  hand,  the  release  management  teams  should  optimize  for  delivering  against 
clearly  defined  requirements,  ability  to  have  longer  release  cycles,  which  depends  on  being  free  from 
production  firefighting. 

To  define  the  acceptance  contract,  have  the  most  senior  persons  from  both  the  production  and  release  manage¬ 
ment  teams  define  how  pre-production  builds  should  be  deployed  into  production.  Critical  questions  include: 

•  Who  designs  and  specifies  the  production  environment  requirements? 

•  At  what  point  does  operations  get  involved? 

•  What  is  the  nature  of  the  hand-off  between  release  management  and  production? 
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•  Who  is  responsible  for  creating  build  criteria  and  scheduling  build  deployments? 

•  What  are  the  build  dependencies?  For  example,  for  Java  Virtual  Machines  (JVM),  which  versions  and  vendors 
will  be  officially  supported?  How  will  unsupported  JVMs  be  transitioned  to  supported  JVMs? 

•  What  hardware  is  on  production  and  development  systems?  Can  the  build  systems  support  disparate 
systems?  (For  example,  certain  lower  cost  Intel  server  platforms  are  notorious  for  switching  components 
weekly,  undermining  the  ability  to  have  uniform  builds  due  to  driver  requirements.) 

•  What  are  the  roles  of  machines  used  by  release  management?  (For  example:  test,  staging,  production, 
developer  toys,  etc.) 

•  Will  release  engineers  have  access  to  production  systems?  (Incidentally,  this  is  strongly  discouraged  and  is 
discussed  in  the  next  section.) 

•  Do  all  release  plans  have  a  valid  back-out  plan  before  being  accepted  for  deployment  into  production?  (Of 
course,  the  CAB  will  oversee  and  enforce  this  as  well.) 

Moving  From  Production  Acceptance  To  Deployment 

Remember,  the  release  engineers  design  the  builds.  Operations  staff  must  accept  and  then  use  the  build  process 
to  provision  infrastructure  into  production.  Provisioning  infrastructure  into  production  will  involve  the 
following  steps: 

1.  The  release  management  team  must  first  have  checked  in  all  of  the  necessary  build  tools,  software  media 
and  documentation  to  the  DSL.  Production  staff  will  check  out  the  builds  from  the  DSL,  which  will  be  used 
to  build  and  deploy  the  production  infrastructure. 

2.  Remember  that  the  team  responsible  for  building  new  infrastructure  has  separation  of  duty  requirements. 
For  example,  no  developers  should  be  part  of  the  build  process,  both  for  security  reasons  and  to  advance 
the  goal  of  the  operations  staff's  ability  to  provision  and  maintain  infrastructure  unaided. 

3.  The  operations  team  will  provision  the  system.  Optionally,  a  QA  team  may  first  test  that  the  system  is 
actually  stable  and  functional,  or  it  may  be  deployed  into  production  on  a  limited  basis  on  low-impact  and 
non-essential  services,  or  a  combination  of  the  two. 

4.  The  operations  staff  will  submit  an  RFC  into  the  CAB  to  get  approval  and  a  schedule  for  system  rollout. 

5.  Once  the  deployment  is  approved  and  scheduled,  it  is  released  into  production. 

Define  Production  Plan  For  Patching  And  Release  Refresh 

Despite  the  urgency  attached  to  applying  software  patches,  patch  deployment  ideally  belongs  in  the  release 
management  process.  The  "patch  and  pray"  phenomenon  is  well-documented;  it  refers  to  the  fact  that  neither 
patching  nor  avoiding  patching  seems  to  achieve  the  objectives  of  creating  an  available  and  secure  computing 
platform.  We  observe  that  high-performing  IT  operations  organizations  patch  far  less  frequently  than  typical  IT 
organizations,  and  yet  they  still  achieve  their  desired  security  posture.  It  is  incorrect  to  assume  that  they  do  this  at 
the  expense  of  security!  Rather,  they  effectively  manage  risk  and  use  compensating  controls  instead  of  patching. 
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Strive  to  apply  and  test  patches  on  the  pre-production  systems  before  deploying  changes  into  production.  Ensure 
that  the  pre-production  and  production  systems  stay  in  sync  to  the  extent  possible.  While  production  and  pre- 
production  hardware  may  be  radically  different  (e.g.  high-end  vs.  low-end  platforms),  configuration  variance 
can  be  managed  so  that  testing  and  qualifying  changes  (including  patches)  can  be  effective.  Creating  repeat- 
able  builds  for  test  systems  allows  for  the  constant  "blowing  up  of  boxes"  to  test  new  configurations,  while  guaran¬ 
teeing  that  a  clean  copy  of  the  pre-production  system  exists  that  is  100%  in  synch  with  that  in  production.11 

Due  to  the  complexity  of  modern  systems,  patching  in  production  can  easily  create  errors,  both  immediate 
and  latent.  Latent  errors  can  accumulate,  increase  configuration  variance,  create  system  errors,  and  may  even 
compromise  security.  By  moving  patching  into  the  pre-production  test  environment  and  the  release 
management  process,  you  are  more  likely  to  catch  errors  due  to  the  better  control  and  testing. 

Ideally,  we  want  to  create  a  process  where  systems  are  being  provisioned  into  production  in  a  planned  and 
scheduled  manner  instead  of  always  being  deployed  in  an  urgent  and  ad  hoc  manner.  To  do  this,  we  will  create 
a  process  that  evaluates  the  patches  and  issues  from  problem  management,  as  well  as  security  and  update 
bulletins  from  subject  matter  experts  (e.g.  CERT,  SEI,  SANS,  vendors,  etc.)  for  applications  in  the  DSL. 

Consider  evaluation  questions  such  as: 

•  Is  this  a  material  threat  to  our  ability  to  deliver  safe  and  reliable  service  to  the  business? 

•  Can  we  mitigate  this  threat  without  applying  the  patch  or  update? 

•  Can  we  test  the  impact  of  the  update  and  feel  confident  that  our  tests  will  predict  the  outcome  on  our 
production  systems? 

•  When  is  the  next  release  cycle?  Can  we  package  this  update  with  other  tested  updates? 

•  If  we  have  to  do  this  now,  how  can  we  minimize  the  risk  of  unexpected  consequences? 

•  If  we  cannot  reduce  the  risk  of  exposure  through  testing,  and  we  cannot  bundle  this  with  any  other  releases, 
then  can  we  get  the  stakeholders  and  IT  management  to  sign  off  on  the  risk? 

•  Create  a  release  schedule  that  achieves  the  above  objectives,  attempting  to  bundle  patches  and  updates  into 
releases  instead  of  applying  individual  patches  to  machines.  Obviously,  each  of  these  releases  needs  to 
involve  the  CAB  for  review  and  approval  prior  to  deployment.  We  want  to  keep  the  release  management 
team  focused  on  longer-lifecycle  projects,  which  get  deployed  on  a  regular  basis.  The  longer  the  shelf  life  of 
the  production  builds,  the  more  stable  the  production  infrastructure  will  be. 

Close  The  Loop  Between  Production  And  Pre-Production 

Before  we  entered  this  phase,  we  started  to  manage  the  change  process  to  avert  risky  changes.  Now,  as  we  bring 
the  release  engineering  capabilities  to  bear  on  the  problem,  managing  and  documenting  changes  become  even 
more  important.  In  order  to  maximize  the  post-deployment  configuration  integrity  and  shelf-life  of  the 
production  builds,  all  production  changes  must  be  reflected  in  the  new  builds,  lest  they  be  overwritten  by  a 
new  software  rollout.  This  is  to  ensure  that  when  infrastructure  is  replaced  or  rebuilt,  it  is  replaced  by  systems 
that  are  functionally  identical  to  the  original  systems.  If  you  are  treating  production  infrastructure  like  fuses, 
success  hinges  on  the  production  and  stored  fuses  being  identical. 

One  step  you  can  take,  which  is  actually  required  in  most  regulated  industries,  is  to  prevent  developers  and 
release  management  engineers  from  accessing  production  infrastructure.  The  notion  that  "those  who  built  the 


"The  NIST  has  a  reference  document  about  security  patching  at:  http://csrc.nist.gov/publications/nistpubs/800-40/sp800-40.pdf. 
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airplane  are  not  allowed  to  fly  it"  is  called  "separation  of  duty"12  and  is  required  for  security  reasons  to  ensure 
that  no  single  person  can  introduce  uncontrolled  production  changes.  For  the  same  reason,  we  recommend 
separating  duties  in  IT  despite  the  cultural  difficulties  it  poses  (i.e.  it  is  often  difficult  to  take  away  root  access 
from  developers,  but  we  have  found  that  having  an  availability  level  greater  than  95%  is  difficult  without 
having  clear  separation  of  roles). 

To  reiterate,  to  be  certain  that  production  builds  stay  in  synch  with  the  golden  builds,  use  a  detective  control 
mechanism  to  ensure  build  integrity.  These  days,  systems  have  tens  of  thousands  of  files,  and  hundreds,  or 
even  thousands,  of  configuration  options,  and  countless  file  versions.  Without  some  automated  way  of 
scanning  for  changes,  it  is  far  too  easy  to  make  a  change  (accidentally  or  not)  and  then  not  record  it.  Only 
through  the  automated  detection  of  changes  can  we  close  the  loop  between  production  and  pre-production. 

What  You  Have  Built  And  What  You  Will  Likely  Hear 

Successfully  implementing  this  phase  will  typically  reduce  the  amount  of  unplanned  work  down  to  15%  or 
less.  By  drastically  reducing  configuration  counts,  we  can  significantly  change  the  staffing  allocation  from 
unplanned  to  planned  work,  and  consequently  increase  the  server/sysadmin  ratio. 

"The  single,  largest  improvement  an  IT  organization  can  benefit  from  is  implementing  repeatable  system 
builds.  Being  a  systems  administrator  at  heart,  /  cannot  stress  how  much  this  will  positively  influence  any 
systems  management  organization.  But  you  can't  do  this  without  first  managing  change  and  having  an 
accurate  inventory.  When  you  convert  a  person-centric  and  heavily  manual  process  to  a  quick  and 
repeatable  mechanism,  the  reaction  is  almost  always  very  positive. 

Using  a  defined  image  for  a  fumpstart,  Kickstart,  disk  clone,  Ghost,  or  any  other  automated  installation 
system  will  immediately  decrease  the  error-prone,  manually  intensive  system  setup  and  configuration 
process.  Even  partially  automated  release/build  processes  greatly  improve  the  ability  for  individuals  and 
organizations  to  be  freed  from  firefighting  and  to  focus  on  their  areas  of  real  value  and  expertise.  And  by 
making  it  more  efficient  to  rebuild  than  repair,  you  also  get  much  faster  systems  recovery  and  significantly 
reduced  downtime."- joe  judge,  former  iso,  adero,  inc. 

"Before  the  application  of  change  management  processes  and  controls  on  our  release  management  processes, 
we  lacked  optimal  control  over  individual  instances  of  production  systems.  Each  system  was  at  some  level 
unique  and  would  behave  in  unique  ways.  Recreating  systems  to  exactly  the  same  specifications  was  not 
always  possible.  Now,  we  are  able  to  record  every  detail  of  a  build  with  100%  certainty  and  ensure  that 
production  systems  remain  identical  to  our  golden  builds.  This  enables  us  to  rapidly  remediate  problems 
with  confidence.  This  is  yet  an  additional  benefit  that  comes  from  change  management.  Good  change 
management  results  in  achieving  the  ability  to  maintain  known  good  configurations  which  can  be  applied 
to  remediate  failed  changes  or  other  system  failures."— stive  darby,  vp  of  operations,  ip  services,  inc. 

"We  can  identify  changes  immediately  and  match  them  up  with  documented  planned  changes.  We  have 
documentation,  testing  procedures  and  back  out  plans  required  in  the  change  management  process.  This 
allows  us  to  quickly  rebuild  and  identify  what  changed  and  the  impact  it  had  to  our  systems.  Time  is  a 
precious  commodity  and  with  our  tools  we  our  managing  our  time  more  effectively  and  efficiently."— karen 
fragale,  data  center  manager 


12  Also  known  as  “segregation  of  duties". 
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Benefits  arising  from  this  phase  are: 

•  We  have  created  a  release  engineering  team  with  well-defined  roles  and  responsibilities,  to  enable  an  effective 
working  relationship  with  the  production  team. 

•  We  have  created  a  process  for  the  release  engineering  team  to  define  and  generate  infrastructure  that  can  be 
repeatedly  built. 

•  We  have  further  decreased  uncontrolled  production  changes,  which  increases  the  amount  of  time  available 
to  work  on  planned  tasks. 

•  We  have  created  a  new  problem  resolution  mechanism,  making  it  cheaper  to  rebuild  than  to  repair.  This 
creates  an  alternative  to  protracted  fire-fighting. 

•  We  have  enabled  shifting  of  senior  staff  from  the  front-lines  to  the  release  management  area,  where  the 
defect  repair  costs  are  lowest.  This  further  enables  junior  staff  to  handle  ever  more  challenging  production 
responsibilities. 

•  We  have  moved  more  staff  off  the  front  lines  to  work  earlier  in  the  lifecycle. 

•  We  have  closed  the  loop  between  production  and  release  management,  to  curb  production  configuration 
variance. 

•  We  have  enabled  the  continual  reduction  of  unique  configurations  in  deployment,  increasing  the  mastery  of 
each  configuration  in  deployment,  increasing  the  server/sysadmin  ratio. 

•  We  have  mitigated  the  "patch  and  pray"  dilemma,  by  integrating  software  updates  into  the  release 
management  processes,  where  patches  can  be  safely  tested  and  rolled  out. 

Audit  Tips: 

•  Fully  document  the  build  process  from  feature  request,  to  build  definition,  to  build  acceptance. 

•  Fully  document  the  acceptance  and  handoff  process  between  the  pre-production  and  production  teams. 

•  Prepare  reports  on  production  rollouts  of  software,  change  success  rate,  time  required  to  complete  the 
rollout,  and  the  integration  with  the  change  management  processes. 

•  Document  the  process  of  how  software  is  evaluated,  accepted  into,  and  purged  out  of  the  DSL. 

•  Generate  a  report  of  the  percentage  of  deployed  systems  that  match  the  golden  builds. 

•  Document  the  process  used  to  track  threats  and  generate  projects  in  the  release  management  processes 
for  patch  updates  and  software  rollouts. 

•  Document  the  policies  for  the  clean-room  build  process. 

•  Be  able  to  show  how  systems  are  certified.  In  other  words,  "How  do  I  know  that  what  I  built  is  what  I 
intended  to  build?" 

•  Be  able  to  provide  a  list  of  all  exceptions  to  the  golden  builds  and  justifications  for  them.  An  abundance 
of  unexplained  exceptions  is  evidence  of  an  ineffective  process. 
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Phase  Four:  Continual  Improvement 

" A  vision  without  a  task  is  but  a  dream.  A  task  without  a  vision  is  but  a  drudgery;  but,  a  vision  and  a  task 
are  the  hope  of  the  world."— unknown  (found  on  wall  of  church  outside  Sussex  England,  circa  i700> 

Philipp  M.  Nattermann  wrote  in  the  McKinsey  Quarterly  that  if  all  you  are  doing  is  adopting  best  practices, 
then  eventually,  all  you  are  going  to  get  is  competitive  parity.13  In  order  to  really  excel,  you  need  to  optimally 
apply  all  your  resources  to  achieve  the  real  business  goals.  To  do  this,  we  want  to  make  sure  that  we  are  best 
applying  resources  towards  the  accomplishment  of  objectives.  In  a  world  where  competitive  business  and 
security  pressures  keep  increasing,  the  only  way  to  effectively  respond  is  to  stabilize  the  environment  and 
reduce  firefighting.  This  results  in  more  time  available  to  address  the  needs  of  the  organization.  One  of  the  best 
ways  to  manage  and  allocate  resources  during  this  continuous  improvement  journey  is  with  metrics. 

Metrics  And  How  To  Use  Them 

" The  better  you  get,  the  better  you'd  better  get."— david  allen 

A  management  truism  is  that  you  cannot  manage  what  you  cannot  measure.  In  the  case  of  IT  and  automated 
systems,  it  is  far  too  easy  to  start  collecting  a  mountain  of  data  and  generate  metrics  that  are  of  little  or  no 
value,  regardless  of  how  much  you  analyze  them.  In  this  phase,  we  will  focus  on  metrics  that  aid  decision 
making  and  give  indicators  as  to  the  qualitative  type  of  work  being  done  (i.e.  planned  vs.  unplanned,  active 
vs.  reactive,  early  vs.  late  in  the  lifecycle,  and  so  on).  We  will  discuss  the  types  of  metrics  we  can  generate,  and 
describe  how  to  use  them  to  guide  improvement  efforts. 

In  general,  the  key  metrics  for  IT  operations  are  the  availability  metrics,  such  as  the  ITIL  resolution  process 
metrics  of  MTTR  and  MTBF.  The  problem  is  that  virtually  all  the  factors  that  affect  these  metrics  live  in  the 
controls  and  release  process  areas.  Essentially,  think  of  MTTR  and  MTBF  as  being  symptomatic  of  decisions 
made  elsewhere. 

Upon  completion  of  the  first  three  Visible  Ops  phases,  we  have  created  fully-functional  release,  controls  and 
resolution  processes  as  well  as  having  created  a  closed-loop  feedback  mechanism  between  all  of  the  three  process 
areas.  In  fact,  we  have  done  more  than  that:  We  have  created  a  minimal  closed-loop  system  that  is  capable  of 
improving  itself.  What  do  we  mean  by  this?  By  completing  the  previous  three  phases,  we  can  now  generate 
metrics  for  the  three  key  process  areas  (release,  controls  and  resolution)  that  dictate  the  following: 

•  Release— How  efficiently  and  effectively  do  we  generate  and  provision  infrastructure? 

•  Controls— How  effectively  do  we  make  good  change  decisions  that  keep  production  infrastructure  available, 
predictable  and  secure? 

•  Resolution— When  things  go  wrong,  how  effectively  do  we  diagnose  and  resolve  issues? 


B  Nattermann,  Philipp  M.  McKinsey  Quarterly  2000.  Reprinted  in  BetterManagement.com 
http://www.bettermanagement.com/Library/Library.aspx?a=ll&LibraryID=8387 
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We  can  generate  metrics  from  these  three  process  areas,  and  use  them  to  guide  investment  in  future  improve¬ 
ment  efforts.  After  all,  the  goal  of  the  first  three  phases  was  to  stabilize  the  environment  and  free  up  enough 
resources  to  work  on  proactive  projects  that  deal  with  root  causes.  With  those  phases  complete,  we  have  laid 
the  groundwork  to  focus  on  continuous  improvement.  Our  goal  will  be  to  use  the  analysis  of  metrics  to 
achieve  the  following  high-level  objectives: 

•  Increase  the  amount  of  resource  and  staff  working  on  process  areas  early  in  the  IT  operations  lifecycle,  where 
defect  repair  costs  are  the  lowest.  In  other  words,  move  key  staff  into  pre-production  engineering  roles. 

•  Increase  the  amount  of  time  spent  on  proactive  and  planned  work,  instead  of  reactive  and  unplanned  work. 

•  Increase  organization  productivity  by  increasing  change  rates,  change  success  rates,  and  the  business  value 
of  changes. 

•  Keep  closing  the  loop  and  using  detective  controls  to  carefully  reduce  variance  including  configuration 
variance,  variance  between  planned  work  and  actual  work,  and  variance  between  builds. 

Note  that  we  focus  not  on  service  levels,  but  on  the  qualitative  nature  of  the  work  being  done.  This  is  because 
service  levels  are  a  symptom  of  the  quality  and  efficiency  of  an  IT  organization.  Below,  we  list  sample  metrics 
for  each  of  the  three  key  process  areas,  followed  by  an  example  of  some  of  the  general  improvement  projects 
that  you  can  take  on  to  achieve  the  goals  stated  above. 

Release  Metrics 

•  Time  to  provision  known  good  builds— How  long  does  it  take  to  build  and  provision  infrastructure  from 
bare-metal?  (Shorter  is  better,  and  should  be  shorter  than  any  MTTR  requirement.) 

•  Number  of  turns  to  a  known  good  build — How  many  times  must  the  build  be  modified  before  it  is  accept¬ 
able  for  deployment?  (Lower  is  better.  A  high  number  indicates  the  need  for  a  more  automated  process.) 

•  Shelf  life  of  builds— How  long  will  each  build  be  in  production  until  it  is  replaced?  (Longer  is  typically 
better,  because  it  enables  release  management  teams  to  stay  out  of  reactive  mode.) 

•  Percent  of  systems  that  match  known  good  builds— According  to  the  detective  controls,  how  many 
production  systems  actually  match  their  corresponding  golden  builds?  (Higher  is  better,  because  it  indicates 
absence  of  uncontrolled  production  configuration  drift.) 

•  Percent  of  builds  that  have  security  sign-off— How  many  configurations  were  approved  by  security? 
(Higher  is  better,  because  it  indicates  that  security  is  involved  in  the  standard  "blessing"  process.) 

•  Number  of  fast-tracked  builds— How  many  builds  were  rushed  into  production  via  the  emergency  change 
process?  (Lower  is  better,  because  each  of  these  represent  a  deviation  from  intended  process.) 

•  Ratio  of  release  engineers  to  system  administrators— What  percentage  of  staff  is  deployed  on 
pre-production  processes?  (Higher  is  typically  better  because  the  cost  of  defect  repair  is  much  lower 
in  pre-production.) 

Controls  Metrics 

•  Number  of  changes  authorized  per  week— How  many  changes,  as  measured  by  the  change  management 
process?  (In  general,  higher  is  better,  as  long  as  the  change  success  rate  remains  high  as  well.) 

•  Number  of  actual  changes  made  per  week— How  many  changes,  as  measured  by  detective  controls?  (In 
general,  higher  is  better,  but  should  not  be  higher  than  the  changes  authorized  by  the  CAB!) 
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•  Number  of  unauthorized  changes— How  many  changes  circumvented  the  change  process?  This  is  typically 
measured  by  using  the  detective  controls,  or  worse,  through  unplanned  outages.  (Lower  is  better.) 

•  Change  success  rate— How  many  changes  are  successfully  implemented,  without  causing  an  outage  or 
episode  of  unplanned  work.  (Higher  is  better:  Best  in  class  does  better  than  99%.) 

•  Number  of  service-affecting  outages— How  many  changes  result  in  service  impairment  or  an  outage? 
(Lower  is  better.) 

•  Number  of  emergency  changes— How  many  changes  required  using  the  CAB/EC  process.  (Lower  is  typically 
better,  since  it  indicates  a  higher  percentage  of  planned  work.) 

•  Number  of  "special"  changes— How  many  changes,  for  whatever  reason,  are  being  made  outside  of  the 
change  process?  (Lower  is  better,  because  these  indicate  that  a  change  process  is  not  fully  functional,  because 
management  is  allowing  certain  categories  of  changes  to  bypass  change  management  entirely.) 

•  Number  of  "business  as  usual"  changes— How  many  low-impact  changes  were  there? 

•  Change  management  overhead— How  much  effort  (in  hours  or  staffing)  is  the  change  management 
function  consuming?  (In  general,  this  number  should  be  low.  A  high  number  may  indicate  a  bureaucratic 
process,  rather  than  one  that  enables  productivity.) 

•  Changes  submitted  vs.  changes  reviewed— What  is  the  ratio  of  evaluated  change  requests  against  the  total 
change  requests  turned  in? 

Resolution  Metrics 

•  MTTR— Mean  Time  To  Repair:  the  average  time  to  restore  service  after  an  interruption. 

•  MTBF— Mean  Time  Between  Failure:  the  average  time  between  service  incidents. 

Other  Improvement  Points 

Release  Area  Improvement  Points 

•  Enforce  a  standard  build  across  all  similar  devices. 

•  Track  all  configurations  in  use  (Development,  Test,  and/or  Production)  and  ensure  that  there  are  stored  builds 
for  each. 

•  Do  bare-metal  builds  whenever  possible. 

•  Perform  change  audits  on  all  production  systems.  Use  detective  controls  to  assure  that  all  builds  in 
production  tie  to  known  good  builds.  When  they  do  not,  investigate  and  mitigate. 

•  Segregate  the  development,  test  and  production  systems.  Developers  need  systems  they  can  use  without  fear 
of  disrupting  other  activities.  Test  systems  must  reflect  production  configurations  and  be  able  to  be  locked 
down  for  controlled  tests.  Production  systems,  of  course,  must  remain  thoroughly  controlled  in  terms  of 
access  and  the  change  processes  followed. 

•  Capture  the  known  good  state  or  "golden  build"  as  part  of  the  release  management  process. 

•  Create  a  library  of  automated  build  systems  for  all  critical  devices  (i.e.  repeatable,  automated  processes  that 
can  provision/re-provision  all  critical  systems). 

•  Create  a  definitive  software  library  (DSL). 

•  Define  a  process  for  accepting  applications  into,  and  retiring  them  out  of,  the  definitive  software  library  (DSL). 
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•  Confirm  that  all  deployed  system  images  are  under  version  management. 

•  Create  regular  meetings  to  determine  the  relevance  of  the  repeatable  build  inventory. 

•  Confirm  that  repeatable  builds  have  predictable  and  bounded  remediation  times. 

Controls  Improvement  Points 

•  Change  management  meetings  must  have  a  specified  agenda. 

•  Changes  should  be  categorized  (e.g.  business  as  usual,  application  change),  with  appropriate  workflows. 

•  Work  with  audit  to  agree  on  what  reporting  information  should  be  provided,  and  learn  what  they  will  audit 
and  inspect. 

•  If  the  rate  of  change  is  increasing  beyond  control,  investigate  the  causal  factors. 

•  If  the  rate  of  change  is  decreasing,  confirm  that  staff  is  not  circumventing  the  change  process. 

•  Automate  key  change  management  processes. 

•  Create  a  feedback  loop  from  production  to  the  release  engineers.  Often,  engineering  groups  get  out  of  sync 
with  the  "real  world"  in  terms  of  what  works  and  what  doesn't. 

•  Create  and  use  your  CMDB  to  track  production  infrastructure— make  sure  it  is  used  by  as  many  staff  as 
possible  as  a  central  knowledge  base. 

Resolution  Improvement  Points 

•  Internalize  the  fact  that  change,  regardless  of  source,  is  the  root  cause  of  most  remediation  efforts. 

•  Internalize  the  fundamental  relationship  between  MTTR  and  availability.  By  improving  MTTR,  you  also 
improve  overall  availability. 

•  Constantly  improve  problem  diagnosis  processes,  since  this  is  the  longest  part  of  the  repair  cycle. 

•  Define  bulletproof  rollback  processes  to  recover  from  failed  or  unauthorized  changes. 

•  Make  sure  that  the  problem  ticketing  system  can,  for  affected  systems,  show  all  open  work  orders  and 
production  changes,  both  authorized  and  unauthorized. 

•  During  remediation,  use  the  CMDB  and  change  management  systems  to  see  what  has  caused  failures  in 
the  past. 

•  Track  who  made  a  specific  change,  enforcing  accountability  as  well  as  fostering  knowledge  transfer. 

•  Have  a  process  that  maps  all  detected  changes  to  a  valid  business  purpose  or  authorized  work  order.  Have  the 
Change  Advisory  Board  review  this  information  as  part  of  their  regular  meeting. 

•  Identify  a  set  of  change  owners  for  all  critical  systems. 

•  Track  repeat  offenders  who  circumvent  change  management  policies.  Determine  the  best  course  of  corrective 
action,  starting  with  additional  training  and  escalating  up  to  and  including  formal  disciplinary  action. 

•  Review  problem  tickets  during  the  change  management  meetings  in  order  to  identify  required  actions. 

•  All  repairs  made  in  the  production  environment  must  be  mirrored  in  the  preproduction  environment,  CMDB 
and  the  repeatable  build  process. 
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•  Give  systems  and  services  meaningful  names  that  relate  to  the  functionality  they  provide. 

•  Create  an  end  of  shift  audit  process  where  operations  managers  are  held  accountable  for  changes  that 
happened  on  their  watch:  Changes  should  either  be  mapped  to  authorized  work  or  rolled  back. 

A  Caution  About  Automation 

When  it  comes  to  automation,  IT  process  consultants  usually  warn  organizations  not  to  automate  a  new 
process  before  you  have  had  practice  running  it  with  pencil  and  paper.  For  a  historic  example  to  put  this  in 
perspective,  we  can  look  to  the  challenge  of  bringing  quality  to  automotive  manufacturing  in  the  1980s.  In  the 
book  Why  Smart  Executives  Fail,  Sydney  Finkelstein  describes  how  GM  attempted  to  solve  their  quality  issues 
through  robotic  automation.  By  the  end  of  GM  CEO  Roger  Smith's  program,  they  had  spent  $44  billion  to 
build  the  "factory  of  the  future"— enough  to  have  purchased  Toyota  and  Nissan  combined,  but  without 
meeting  their  quality  or  cost  goals.  Ford  President  Phil  Benton  was  not  surprised  and  reflected  that  consistency 
of  practice  must  come  before  automation. 

When  dealing  with  any  IT  operational  processes,  whether  it  is  related  to  change,  configuration,  release  or 
security,  take  heed.  If  you  cannot  run  these  processes  manually,  do  not  attempt  to  automate  it— all  you  will 
do  is  automate  confusion. 

What  You  Have  Built  And  What  You  Will  Likely  Hear 

" Phases  Three  and  Four  of  Visible  Ops  focus  on  creating  a  repeatable  process  and  striving  for  continuous 
improvement.  Visible  Ops  bundles  all  of  the  service  life  cycle  in  these  two  phases,  and  there  is  obviously  a 
lot  more  to  it  than  you  probably  intend  to  cover.  Visible  Ops  is  imbedding  the  four  aspects  of  a  service  life 
cycle  [ Planning/Definition/Implementation/Supporting ]  in  phases  Three  and  Four.  These  phases  merit  attention 
in  mature  organizations  that  are  beyond  the  phases  of  critical  care.  This  is  then  also  the  beauty  of  Visible 
Ops:  it  points  out  to  organizations  where  the  start  point  is,  and  how  to  get  out  of  the  chaos."— jan  vromant, 

ITSM  CONSULTANT 

At  this  point  you  have  created  a  closed-loop  management  system  that  uses  metrics  and  controls  to  improve 
over  time  and  better  react  to  the  environment.  Points  to  consider: 

•  Auditors  hate  pushing  organizations  to  implement  controls,  especially  if  doing  so  creates  grudges  and  literal 
interpretations  of  findings.  Ideal  controls  are  owned  by  the  business  to  meet  objectives  instead  of  being  there 
only  to  generate  positive  audit  findings. 

•  The  closed-loop  management  system  constantly  reinforces  the  culture  of  causality. 

•  Objective  data  backs  up  assertions  of  improvement  and  issues  that  need  to  be  dealt  with. 

•  As  opposed  to  management  by  belief,  you  have  firmly  moved  to  management  by  fact. 
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Summary 

Congratulations!  If  you've  made  it  this  far,  you've  learned  all  about  the  four  phases  of  Visible  Ops.  We  hope 
you've  found  many  practices  that  you  can  quickly  implement  in  your  own  organization.  Moreover,  we  know 
that  many  of  you  will  not  only  implement  these  practices,  but  also  improve  on  them— if  you  are  one  of  these, 
please  make  sure  to  email  us  and  tell  us  about  it! 

Based  on  our  numerous  years  of  research,  we  are  confident  that  if  you  follow  the  steps  outlined  in  this  book,  you 
will  be  able  to  replicate  the  amazing  transformation  that  other  IT  practitioners  have  achieved  with  their 
organizations  to  fix  their  availability  and  security  issues.  We  not  only  hope  that  you  have  found  this  journey  to 
be  useful,  but  that  it  has  illuminated  a  different  way  to  approach  solving  IT  operational  process  issues. 

IT  practitioners  can  use  Visible  Ops  to  evolve  from  its  artisan  roots  to  a  system  of  repeatable  and  verifiable 
processes,  where  security,  availability,  quality  and  value  are  built  into  the  processes,  instead  of  being  inspected 
only  at  the  end.  If  this  sounds  familiar,  there's  a  good  reason.  These  are  the  themes  that  Deming  espoused 
throughout  his  career.  As  IT  practitioners  adopt  a  process-focused  approach,  more  of  Deming's  genius  will  seem 
applicable  on  a  daily  basis.  Here  are  some  of  his  quotable  gems: 

•  "If  you  can't  describe  what  you  are  doing  as  a  process,  you  don't  know  what  you're  doing." 

•  "It  is  not  enough  to  do  your  best;  you  must  know  what  to  do,  and  then  do  your  best." 

•  "Does  experience  help?  No!  Not  if  we  are  doing  the  wrong  things." 

•  "We  should  work  on  the  process,  not  the  outcome  of  the  processes." 

•  "Learning  is  not  compulsory... neither  is  survival." 

High-performing  IT  organizations  clearly  have  a  passion  for  process  and  consistency.  They  integrate  security 
and  IT  operations  just  as  the  automotive  industry  successfully  integrated  quality  into  manufacturing.  These 
organizations  have  created  the  culture  necessary  to  ensure  consistent,  repeatable  and  verifiable  IT  operational 
processes,  where  both  the  security  and  operations  groups  are  motivated  to  detect  and  reduce  operational 
variance.  These  practices  bridge  the  gap  between  security  and  IT  operations,  and  also  ensure  that  they  always 
work  together  to  achieve  common  objectives  and  business  goals. 

It  is  our  sincere  hope  that  the  Visible  Ops  methodology  will  help  your  organization  in  your  process 
improvement  journey. 

Sincerely, 

Kevin  Behr 
Gene  Kim 
George  Spafford 
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Information  Technology  Process  Institute  (ITPI) 

This  handbook  was  developed  by  the  Information  Technology  Process  Institute  (ITPI).  The  ITPI  is  a  not  for 
profit  organization,  engaged  in  three  principle  areas  of  activity:  research,  benchmarking  and  the  development 
of  prescriptive  guidance  for  practitioners  and  business  executives.  The  ITPI  has  collaboration  agreements  in 
place  with  research  organizations  such  as  the  Software  Engineering  Institute  at  Carnegie  Mellon  University  and 
the  Decision  Sciences  program  at  the  University  of  Oregon.  We  are  currently  developing  the  necessary 
guidance  that  solves  the  common  objectives  of  IT  Security,  Corporate  Governance,  Audit  and  Operations. 
Through  research,  development  and  benchmarking,  the  ITPI  creates  powerful  measurement  tools,  prescriptive 
adoption  methods,  and  control  metrics  to  facilitate  management  by  fact. 

For  more  information  please  contact  the  ITPI  at: 

ITPI 

2896  Crescent  Avenue 
Eugene,  Oregon  97408 
Main  Telephone:  (541)  485-4051 
Main  Fax:  (541)  485-8163 
http://www.itpi.org 
info@itpi.org 

The  ITPI  also  runs  the  Community  of  Practice  Listserv  (ICOPL).  The  intent  of  this  email  list  is  to  create  a  forum 
where  we  can  exchange  ideas,  solutions,  works  in  progress,  and  advance  the  cause  of  how  IT  operations, 
security,  audit,  and  management  can  work  together  to  solve  common  objectives.  If  you  are  interested: 

•  To  subscribe:  send  a  blank  (subject  and  body  are  ignored)  e-mail  to  icopl-subscribe@itpi.org. 

•  To  unsubscribe:  send  a  blank  e-mail  to  icopl-unsubscribe@itpi.org. 

•  To  email  the  list,  send  email  to  icopl@itpi.org. 

•  This  page  is  reproduced  at  http://www.itpi.org/home/icopl.php. 
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Append:  Preparing  For  Audit 

For  many  IT  practitioners,  the  entire  audit  function  may  be  one  of  the  most  mysterious  and  misunderstood 
roles  encountered  when  working  in  certain  industries.  Ail  too  often,  auditing  is  viewed  as  a  necessary  evil,  and 
therefore,  should  be  characterized  by  a  confrontational  relationship.  Yet  in  high-performing  organizations, 
there  is  a  mutual  respect  between  the  IT  teams  and  the  internal  and  external  IT  audit  teams  they  work  with. 
In  these  situations,  IT  teams  view  auditors  as  additional  resources  to  ensure  that  appropriate  controls  are  in 
place  and  effective. 

Typically,  painful  audit  findings  reveal  the  absence  of  effective  processes  and  controls  rather  than  being  the 
fault  of  the  auditors.  Just  as  the  manufacturing  world  realized  the  need  for  quality  control  processes,  IT  is 
finally  recognizing  that  processes  and  controls  must  be  implemented  as  well.  When  these  processes  are  well- 
documented,  and  documentation  exists  that  can  demonstrate  the  controls  are  working,  audits  usually  go  much 
more  smoothly,  because  auditors  have  a  readily  identifiable  desired  state  to  audit  against. 

When  processes  are  not  documented,  auditors  must  grade  you  against  their  own  processes.  Worse,  when 
controls  do  not  exist  to  demonstrate  that  the  processes  are  being  followed,  auditors  must  go  into 
"archaeology"  mode  to  determine  for  themselves  if  the  systems  meet  documented  control  objectives.  For 
example,  if  you  claim  to  have  change  management  meetings  but  do  not  have  any  formally  recorded  meeting 
notes,  how  can  auditors  verify  that  the  meetings  actually  happened? 

The  level  of  documentation  must  be  commensurate  with  the  risks  associated  with  the  changes.  A  modification 
to  a  report  heading  is  likely  to  have  minimal  risk  associated  with  it.  A  substantial  re-write  of  the  Enterprise 
Resource  Planning  (ERP)  system  is  a  very  different  matter,  has  much  more  risk  and  thus  requires  additional 
processes  and  proof  that  those  processes  are  effective.  Resist  the  urge  to  document  everything  and  instead 
focus  efforts  on  creating  evidence  that  the  right  processes  are  in  place  and  are  being  followed. 

To  explain  this  further,  we  will  examine  the  way  auditors  view  the  world,  which  is  through  three  broad 
categories  of  controls. 

Controls  101 

Auditors  often  view  the  world  through  the  lens  of  risks  and  controls.  Risks  exist,  and  you  can  mitigate  them 
by  either  preventing  or  detecting  them,  and  you  should  always  be  able  to  make  corrections  and  recover  should 
the  risks  actually  happen.  To  explain  this  better,  here  are  the  three  categories: 

•  Preventive— controls  that  keep  something  from  happening.  For  example,  policy,  separation  of  duty,  and 
authorization  processes  are  all  preventive  controls. 

•  Detective— analytical  controls  that  monitor  activity  and  processes  to  determine  if  the  preventive  controls 
have  failed  or  if  something  is  out  of  compliance.  For  example,  change  monitoring  and  verification  are 
detective  controls. 
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•  Corrective— corrective  controls  restore  the  situation  back  to  the  expected  state.  For  example,  if  a  system  1 
crashes  due  to  a  failed  change,  reloading  all  applications  from  the  last  known  good  image  to  bring  the  system 
back  online  serves  as  a  corrective  control. 

The  combination  of  the  three  types  of  controls  creates  a  system  of  checks  and  balances  to  help  ensure  that  the  ** 
processes,  people,  and  technology  operate  within  prescribed  bounds.  We  provide  two  simple  examples  of  \ 
controls  to  reduce  the  risk  of  financial  fraud  and  uncontrolled  IT  changes. 


Business  Risk 

Preventive  Control 

Detective  Control 

Financial  fraud  within  vendor 
payment  process  by  someone 
creating  a  false  vendor,  and  then 
paying  themselves  with  fake 
purchase  orders. 

Separation  of  duty:  those  who  can  create  vendor 
accounts  cannot  also  issue  payments  to  vendors. 
Authorization:  vendor  payment  requires 
authorization  from  budget  owner. 

Review  payment  authorization  for 
approved  signatures. 

Uncontrolled  or  unauthorized 
changes  being  made  in  the 
production  IT  environment  thus 
jeopardizing  availability,  integrity 
and  security. 

Separation  of  duty:  pre-production  staff  cannot 
access  production  systems,  and  must  submit 
proposed  changes  via  the  change  manage¬ 
ment  process. 

Authorization:  all  changes  are  reviewed  and 
authorized  by  the  CAB. 

Monitor  production  configurations 
for  changes  to  guarantee  that  all 
changes  map  to  an  authorized 
work  order. 

Separation  of  duties  ensures  that  no  single  person  has  complete  unchecked  access  to  do  unauthorized  things. 
Because  lack  of  segregation  can  create  nearly  endless  opportunities  to  commit  fraud,  developers  are  not  M 
allowed  access  to  production  processes  where  they  can  directly  make  changes  in  regulated  environments.  J 
Instead,  they  must  develop  the  code,  then  forward  it  to  testing.  Once  there,  the  operations  team  can  review  J 
the  change,  assess  risks  and  deploy  it  into  production  if  everything  is  acceptable.  (j 

These  days,  many  audit  concerns  are  driven  by  regulatory  compliance  needs  that  are  required  by  the  industry  or  ^ 
that  assure  the  integrity  of  financial  reporting.  At  one  time,  auditors  and  bean  counters  checked  to  see  if  the 
financial  statements  were  correct  by  opening  up  all  of  the  warehouses  and  counting  all  the  "beans."  In  this  way, 
they  could  verify  that  the  financial  statements  matched  what  they  physically  observed. 

However,  even  the  best  auditors  have  finite  time  and  resources.  Instead  of  going  into  the  warehouse  and  1 
counting  beans,  they  go  to  the  bean  counting  machine  and  check  the  controls  to  determine  whether  the  5 
machine  can  be  trusted.  In  most  cases  it  is  best  to  have  a  combination  of  preventive  and  detective  controls.  If  n 
neither  exists,  or  if  they  are  inadequate,  the  auditors  cannot  trust  the  results  of  the  bean  counting  machine.  ’! 
This  is  a  very  bad  thing,  because  it  erodes  their  ability  to  rely  on  anything  the  bean  counting  machine  did, 
and  requires  opening  up  the  warehouses,  and,  guess  what,  counting  the  beans.  In  other  words,  without  [| 
assurance  that  proper  controls  exist,  far  more  scrutiny  is  required  thus  incurring  substantial  costs.  J 

To  create  a  more  productive  working  relationship  with  auditors,  be  able  to  clearly  describe  your  preventive  * 
processes  and  the  detective  controls  that  prove  they  work  as  expected. 

>1 

A  main  premise  of  this  book  is  that  controls  serve  an  important  purpose  to  ensure  that  our  processes  achieve 
the  desired  business  objectives  and  that  controls  are  not  in  place  simply  to  make  generate  positive  audit  findings 
or  to  comply  with  regulations.  After  all,  a  customer  would  not  feel  safe  if  the  restaurant  only  complied  with 
health  codes  just  to  keep  the  inspectors  at  bay.  They  would  be  happier  if  the  restaurant  handled  food  with  care 
to  keep  customers  healthy,  happy  and  improve  their  overall  dining  experience. 
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IT  is  no  different.  An  organization  that  uses  effective  controls  to  improve  their  processes  typically  has  far  better 
availability,  lower  amounts  of  unplanned  work,  better  security,  and  incidentally,  smoother  audits. 

Auditors — Internal  And  External 

During  your  journey,  you  will  encounter  both  internal  and  external  auditors.  Neither  one  are  considered  part 
of  the  management  structure.  In  other  words,  management  makes  decisions,  manages  risks,  and  runs  the 
business.  Audit  ensures  that  risks  are  managed  and  statements  that  management  makes  are  reliable. 

According  to  professional  standards  promulgated  by  the  Institute  of  Internal  Auditors,  the  internal  audit 
function  reports  directly  to  the  Audit  Committee  of  the  Board  of  Directors.  Organizationally,  internal  audit 
staff  may  be  placed  under  the  CEO  or  CFO,  but  they  are  independent  of  business  management.  Typically, 
internal  audit  reports  are  addressed  to  management  and  copied  to  the  Audit  Committee. 

External  auditors,  on  the  other  hand,  are  third  parties  retained  by  management  to  give  an  unbiased  opinion 
of  the  assertions  made  by  management  and  report  to  the  audit  board  and  board  of  directors.  Whereas  internal 
audit  reports  to  the  audit  board,  external  auditors  are  accountable  to  shareholders,  regulators,  and  potential 
investors.  External  auditors  come  in  and  evaluate  the  effectiveness  of  the  controls  that  are  attested  to  being  in 
place  by  the  company's  senior  management.  If  a  weakness  is  found,  they  include  it  in  an  audit  findings  report. 
If  they  find  a  material  weakness,  the  auditors  may  be  required  to  disclose  their  findings  to  the  appropriate 
regulatory  body,  such  as  the  Public  Company  Accounting  Oversight  Board  (PCAOB).14 

The  Sarbanes-Oxley  Act  Of  2002 

In  the  United  States,  the  Sarbanes-Oxley  Act,  which  aims  to  restore  public  confidence  in  financial  reporting, 
has  generated  a  great  deal  of  activity.  The  law  is  broken  down  into  sections  and  section  404  tends  to  generate 
the  most  concern  in  IT  circles. 

Section  404  is  titled  "Management  Assessment  of  Internal  Controls."  The  section  states  that  it  is,  "...the 
responsibility  of  management  for  establishing  and  maintaining  an  adequate  internal  control  structure  and 
procedures  for  financial  reporting."  Since  IT  is  so  very  pervasive  in  organizations  today,  especially  finance,  there 
must  be  appropriate  controls.  As  mentioned  earlier,  it  is  unrealistic  for  auditors  to  inspect  everything.  They 
must  rely  on  the  presence  of  effective  controls  for  security,  availability  and  data  integrity.  Furthermore, 
external  auditors  must  attest  to  the  controls.  A  weakness  in  internal  controls  could  trigger  a  disclosure  to  the 
SEC  that  there  is  a  control  deficiency.  As  investors  would  most  likely  view  these  disclosures  negatively,  a  public 
company  could  see  a  negative  impact  on  its  stock  price  and  the  possibility  of  a  shareholder  lawsuit.  Needless 
to  say,  this  makes  many  companies  very  nervous. 

For  groups  using  the  ISACA's  Control  Objectives  for  Information  and  Related  Technologies  (COBIT),  please 
recognize  that  Visible  Ops  provides  guidance  on  how  to  implement  some  of  the  controls  set  forth  in  COBIT. 
COBIT  is  very  good  at  identifying  what  needs  to  be  done,  but  implementers  often  require  assistance  from  best 
practice  sources  such  as  Visible  Ops  and  ITIL.  You  will  find  that  Visible  Ops  provides  guidance  relating  to 
controls  identified  in  AI6— Manage  Changes,  DS9— Manage  the  Configuration,  and  Ml  Monitor  the  Process. 


14  Please  note  that  materiality  does  not  always  apply.  If  an  opportunity  for  fraud  exists  and  there  is  no  control,  then  that  is  very  serious 
because  fraud  signifies  a  control  weakness.  Whether  $10  is  taken  or  $10,000,  a  compensating  control  must  be  in  put  place. 
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Increasing  Your  "Auditability" 


Auditors  do  not  like  to  see  organizations  that  simply  have  controls  in  place  to  pass  audits.  They  much  prefer 
to  see  organizations  who  embrace  controls  to  improve  the  business.  To  this  end,  they  look  for  documentation 
of  processes  and  proof  that  controls  are  followed.  The  first  three  phases  of  Visible  Ops  covered  these  points: 

1.  Ask  the  auditors  what  they  are  looking  for  before  an  audit.  Ask  them  for  their  audit  objectives,  if  any  pre¬ 
audit  checklists  or  data  will  be  required  beforehand,  what  meetings  are  required,  specific  areas  they  will 
inspect,  and  so  on.  Most  likely,  they  will  explain  what  they  will  be  looking  for,  and  give  you  an  opportunity 
to  find  out  which  processes  and  policies  you  will  need  to  supply  documentation.  Remember,  it  is  better  for 
you  and  easier  for  the  auditors  if  you  can  articulate  against  which  target  you  wish  to  be  measured. 

2.  Again,  asking  questions  in  advance  is  one  of  your  best  means  for  preparing  for  an  audit. 

3.  It  is  better  to  be  prepared  for  an  audit  and  not  need  material  than  to  have  an  audit  and  wish  you 
had  material. 

4.  Make  sure  to  list  your  perceived  risks.  Back  it  up  with  a  list  of  risks  sorted  in  descending  order  with  the 
highest  risks  at  the  top,  along  with  the  controls  you  created  to  mitigate  them. 

5.  Document  your  preventive  controls,  and  have  detective  controls  in  place  to  show  they  work. 

a.  Document  the  change  management  process. 

b.  Use  the  CAB  meeting  minutes  to  show  that  meetings  are  being  attended  and  used  to  manage  change. 

c.  For  each  authorized  change,  document  the  configuration  changes  from  the  detective  controls  to  show 
that  the  changes  made  were  within  the  scope  of  the  work  order. 

d.  File  the  data  collected  about  change  requests  and  make  it  readily  accessible.  In  some  organizations,  all 
of  the  above  information  lives  in  a  physical  three-ring  binder. 

6.  Keep  a  current  and  accurate  asset  inventory  of  hardware  and  software. 

7.  Document  all  internal  audit  procedures  and  the  proof  that  they  are  being  followed.  For  example,  if  your 
policies  state  that  firewall  logs  are  monitored  by  a  system  with  exceptions  reviewed,  then  you  must  have 
proof  of  following  that  policy  through  logs  of  one  form  or  another. 

8.  Document  all  outages  and  unscheduled  downtime  in  the  systems  along  with  corrective  actions  taken. 

9.  Keep  current  documentation  of  all  exceptions  to  policies. 

10.  List  any  security  incidents  along  with  corrective  actions  taken. 

11.  Be  able  to  produce  previous  audit  findings,  analysis  of  the  findings  and  progress  made  against  findings  that 
warranted  corrective  action. 

You  will  immediately  notice  that  it  is  virtually  impossible  to  prepare  for  an  audit  at  the  last  moment.  This  is 
why  you  must  develop  a  process  culture  that  naturally  produces  the  above-listed  elements.  For  example,  the 
documentation  from  change  management  meetings  helps  track  what  is  going  on  and  capture  knowledge.  It  is 
not  simply  there  to  pass  audits.  When  we  discuss  preparing  for  an  audit,  we  are  highlighting  the  minimum 
that  auditors  will  want  to  see  in  the  form  of  electronic  or  manually  captured  data.  Again,  the  internal  and 
external  audit  groups  can  best  tell  you  what  they  need  to  perform  audits. 
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The  following  summarizes  our  audit  tips  from  each  of  the  first  three  phases: 

Phase  I:  Stabilize  The  Patient  and  Modify  First  Response 

•  Avoid  at  all  costs  creating  an  adversarial  relationship  with  auditors.  Instead,  demonstrate  that  you  have 
effective  management  and  control  processes  in  place,  and  the  documentation  to  prove  it.  If  you  cannot  show 
intended  and  actual  activities,  auditors  go  into  "archaeology"  mode.  (The  worst  thing  you  can  do  is  become 
defensive  and  adversarial,  especially  if  material  control  weaknesses  do  indeed  exist.) 

•  Make  sure  you  have  an  up-to-date  document  describing  your  change  management  process.  Show  this  to 
auditors  up  front  to  illustrate  what  you  want  to  be  measured  against.  Without  it,  they  will  bring  in  their  own 
processes  to  measure  you. 

•  Take  good  meeting  minutes  during  the  CAB  meetings  and  file  them.  Make  sure  they  are  dated.  Showing 
meeting  minutes  to  auditors  to  demonstrate  that  the  meetings  are  actually  taking  place. 

•  The  mantra  of  post-Enron  auditors  is,  "If  it's  not  documented,  it  doesn't  exist."  Therefore,  be  sure  to  docu¬ 
ment  both  your  work  and  your  meetings.  The  correct  level  of  documentation  should  be  commensurate  with 
the  level  of  risk  associated. 

•  To  show  that  your  change  management  processes  function,  meeting  minutes  should  show: 

-  Newly  authorized  and  scheduled  change  requests. 

-  Acceptance  of  implemented  changes,  showing  correlation  between  detected  changes  and  implemented 
changes,  showing  successful  implementation,  acceptance  by  a  change  manager  and  closure  of  the  work  order. 

-  Changes  to  production  equipment  tracked  in  work  logs/work  order  tickets.  These  should  identify  the 
date,  time,  implementer  and  system  along  with  details  of  the  changes  made. 

•  Assemble  a  list  of  changes  made  outside  of  the  change  management  policy  and  corrective  actions  taken. 

•  On  a  regular  basis,  create  and  review  a  report  with  the  number  of  changes  requested,  changes  approved, 
MTTR  and  Change  Success  Rate  by  asset,  functional  area  and  organization,  etc. 

•  Engineer  the  change  workflow  and  ticketing  systems  in  such  a  way  that  "closing”  a  request  or  ticket  is  not 
possible  until  it  has  been  reviewed  and  accepted  by  the  change  manager.  This  ensures  accountability, 
visibility  and  fact-based  management,  instead  of  belief-based  or  faith-based  management. 

•  By  doing  the  above,  you  prove  that  you  have  functioning  preventive,  detective  and  corrective  controls  in 
place.  For  more  information,  refer  to  Appendix  A. 
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Phase  2:  "Catch  &  Release"  and  Find  Fragile  Artifacts  Projects 

•  Be  able  to  show  the  process  of  how  you  generated  the  catalog  of  services  and  the  assets  that  support  them.  ■ 
Remember,  the  inventory  should  include  both  hardware  and  software. 

•  Show  that  you  understand  the  business  processes  you  are  supporting  by  working  with  senior  management  • 
to  rank  the  services  by  importance  to  the  organization  and  their  degree  of  fragility. 

•  Show  how  you  assure  that  the  inventory  is  maintained  and  accurate.  The  lack  of  an  accurate  inventory  may 
indicate  to  auditors  that  there  are  inadequate  controls. 

•  Show  the  list  of  fragile  artifacts  resulting  from  this  phase  as  evidence  that  you  are  performing  risk  mitigation. 

•  Be  able  to  document  the  systems  and  processes  used  to  detect  changes. 

Phase  3:  Create  A  Repeatable  Build  Library 

•  Fully  document  the  build  process  from  feature  request,  to  build  definition,  to  build  acceptance. 

•  Fully  document  the  acceptance  and  handoff  process  between  the  pre-production  and  production  teams. 

•  Prepare  reports  on  production  rollouts  of  software,  change  success  rate,  time  required  to  complete  the 
rollout,  and  the  integration  with  the  change  management  processes. 

•  Document  the  process  of  how  software  is  evaluated,  accepted  into,  and  purged  out  of  the  DSL. 

•  Generate  a  report  of  the  percentage  of  deployed  systems  that  match  the  golden  builds. 

•  Document  the  process  used  to  track  threats  and  generate  projects  in  the  release  management  processes  for 
patch  updates  and  software  rollouts. 

•  Document  the  policies  for  the  clean-room  build  process. 

•  Be  able  to  show  how  systems  are  certified.  In  other  words,  "Flow  do  1  know  that  what  I  built  is  what  I 
intended  to  build?" 

•  Be  able  to  provide  a  list  of  all  exceptions  to  the  golden  builds  and  justifications  for  them.  An  abundance  of 
unexplained  exceptions  is  evidence  of  an  ineffective  process. 

Auditor  Red  Flags  And  Indicators 

Clearly,  auditors  are  concerned  about  the  health  of  the  IT  systems  hosting  applications  that  the  organization 
relies  on.  Thus,  they  focus  on  poor  service  levels  and  unusually  high  velocity  of  change  as  "red  flags" 
indicating  that  there  are  inadequate  controls.  This  is  the  profound  concept  that  came  from  our  study  of  high- 
performing  organizations.  The  controls  that  give  you  good  service  levels  are  exactly  the  same  controls  that 
auditors  look  at  to  mitigate  risks. 
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Appendix  !  The  Information  Technology  Infrastructure  Library  (ITIL)15 

Many  executives  express  frustration  as  they  attempt  to  reign  in  the  chaos  and  expense  associated  with  their  IT 
investments  but  find  little  in  the  way  of  substantive  guidance.  The  IT  Infrastructure  Library  (ITIL)  has  emerged 
as  the  worlds  most  widely  accepted  approach  to  the  management  and  delivery  of  IT  Services. 

If  you  have  not  heard  of  ITIL,  do  not  be  surprised.  ITIL  currently  has  over  42,000  certified  consultants, 
primarily  in  Europe  and  Canada,  with  only  a  small  fraction  of  those  certified  professionals  residing  or 
practicing  in  the  U.S. 

The  Information  Technology  Infrastructure  Library  (ITIL)  represents  a  drastically  different  approach  to  IT  by 
framing  all  activity  under  two  broad  domains  named  "Service  Management"  and  "Service  Delivery" 
respectively.  By  focusing  on  the  critical  business  processes  and  disciplines  needed  to  deliver  services  around  IT, 
the  ITIL  provides  a  maturity  path  for  IT  that  is  not  based  on  technology.  This  accessibility  allows  senior 
executives  to  both  sponsor  and  shepherd  IT  quality  improvement  efforts.  The  ITIL  has  become  the  most  widely 
accepted  approach  to  IT  service  management. 

ITIL  provides  a  comprehensive,  consistent  volume  of  best  practices  drawn  from  the  collective  experience  and 
wisdom  of  thousands  of  IT  practitioners  around  the  world.  By  defining  IT  quality  as  the  level  of  alignment 
between  the  actual  services  delivered  and  the  actual  needs  of  the  business  the  library  serves  as  a  common  point 
of  engagement  for  IT  and  the  other  business  units. 

What  ITIL  Covers 

To  codify  and  organize  the  guidance  contained  in  the  ITIL,  the  British  Standards  Institute  published  the 
BS15000  as  a  code  of  IT  Practice  for  IT  Service  Management.  The  BS15000  organizes  all  of  the  guidance  from 
the  ITIL  into  five  distinct  categories.  The  areas  include:  Service  Design  and  Management  Processes,  Supplier 
Processes,  Resolution  Processes,  Control  Processes,  and  Release  Processes. 


15  Adopted  from:  Behr,  Kevin  and  Kim,  Gene.  “Why  You  Need  To  Know  About  ITIL*."  The  original  article  was  previously  published  and  is 
available  at  http://www.bettermanagement.com/library/library.aspx?libraryid=5711&pagenumber=l. 
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Figure  7:  The  BS  15000  Diagram 


•  Release  Process— This  process  area  answers  the  question  "Where  does  infrastructure  come  from  before  it  is 

deployed?"  This  includes  activities  such  as  the  planning,  designing,  building,  and  configuring  of  hardware  and  1 
software.  Unfortunately,  release  processes  are  traditionally  the  last  process  area  that  organizations  invest  in.  I 
Yet  this  is  the  process  area  that  delivers  the  highest  return  on  investment,  because  it  encompasses  the  entire  j 
pre-production  infrastructure,  where  the  cost  of  defect  repair  is  lowest.  )| 

•  Control  Processes— This  process  area  covers  maintaining  production  infrastructure,  not  only  to  prevent 
service  interruptions,  but  also  to  efficiently  deliver  IT  service.  This  is  done  through  change  management,  as 
well  as  asset  and  configuration  management.  BS  15000  defines  change  management  as  well  as  asset  and 
configuration  management  as  primary  controls.  As  Stephen  Katz,  former  CISO  of  Citibank,  once  said,  I 
"Controls  don't  slow  the  business  down;  like  brakes  on  a  car,  controls  allow  you  to  go  faster." 


•  Resolution  Processes— This  process  area  is  triggered  when  production  infrastructure  does  go  down,  service  j 
is  interrupted,  or  there  is  a  security  issue.  Incident  management  owns  the  customer  relationship,  and  j 
problem  management  owns  the  tasks  of  turning  each  problem  into  a  known  error  that  can  be  more  | 
efficiently  resolved  the  next  time  it  happens.  Ail  too  often,  organizations  that  spend  too  much  time  1 
firefighting  are  unable  to  spend  time  in  the  previous  two  process  areas. 


•  Relationship  Processes— This  area  focuses  on  the  processes  necessary  to  support  effective  customer  relations  as  , 
well  as  the  management  of  third  party  vendors  from  a  performance  and  contractual  standpoint. 

•  Service  Delivery  Processes— The  goal  of  these  processes  is  to  provide  the  best  possible  service  levels  to  meet  ; 
the  business  needs  of  the  organization.  This  process  area  includes  the  monitoring  and  management  of  IT  j 
infrastructure  as  it  relates  to  Security  Management,  Availability  and  Contingency  Management,  Capacity  j 
Management,  Financial  Management  and  Service  Level  Management  and  Reporting. 
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ITIL  Success  Stories 

Now  that  you  have  been  exposed  to  the  ITIL  and  BS15000,  the  next  logical  question  is  "has  anyone  successfully 
adopted  this  framework?"  The  answer  is  a  resounding  yes.  Over  the  last  ten  years  organizations  ranging  both 
in  size  and  industry  have  successfully  integrated  the  best  practices  guidance  and  taken  their  results  to  the 
public.  Some  of  the  most  familiar  names  include  Procter  and  Gamble,  Guinness,  IBM,  British  Airways  and  the 
Internal  Revenue  Service.  Procter  and  Gamble  credit  their  adoption  of  ITIL  practices  with  saving  the  company 
well  over  one  hundred  million  U.S.  dollars  each  year.  Other  organizations,  such  as  Shell,  have  leveraged  the 
ITIL  and  saved  both  large  amounts  of  man-hours  and  hard  currency. 

It  has  become  clear  that  focusing  on  the  development  of  a  process-driven  IT  organization  can  yield  significant 
efficiency  related  savings.  Effectiveness  of  services  delivered  has  also  ranked  highly  as  one  of  the  many  positive 
outcomes  of  best  practice  adoption.  Moving  into  the  realm  of  IT  Service  Management  requires  the  attention 
and  focus  of  the  IT  organization  as  well  as  from  many  important  stakeholders  in  other  business  units. 

Easy  Steps  to  Get  Value  with  ITIL 

How  do  you  get  started  with  ITIL?  After  all,  ITIL  includes  volumes  upon  volumes  of  manuals  that  sometimes 
read  like  a  dictionary.  A  better  approach  would  be  to  look  at  several  key  process  areas  in  your  organization,  to 
assess  where  you  are  versus  best  practices,  and  to  generate  smaller  projects,  bootstrapping  controls  and 
processes  that  need  attention. 

Below,  we  include  a  short  questionnaire  in  several  key  areas  that  have  plenty  of  leverage,  and  a  strong 
probability  of  quick  improvements  with  high  visibility  of  return.  Use  the  questionnaire  to  discover  issues  that 
commonly  cause  widespread  service  disruption  and  firefighting  behaviors. 

Control  processes 

1.  Do  your  system  administrators  spend  too  much  time  fire-fighting? 

2.  Do  you  have  a  well  documented  change  management  process  that  provides  visibility  and  control  points  for 
changes? 

3.  Are  the  largest  percentages  of  problems  caused  by  changes  made  internally  as  opposed  to  externally? 

4.  Can  you  quickly  discover  unauthorized  or  undocumented  changes? 

5.  Can  you  lock  down  production  servers  so  no  change  is  allowed? 

6.  Can  you  map  systems  changes  to  a  change  request  or  an  authorized  work  order? 

7.  Do  you  schedule  all  system  changes  to  fixed  maintenance  windows,  mitigating  the  risk  of  changes? 

8.  Do  you  have  an  end-of-shift  audit  process,  assuring  that  the  operations  manager  is  handing  over  the  data 
center  in  the  same  condition  that  it  was  received? 
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Release  Processes 

1.  Can  you  enforce  a  standard  configuration  build  across  all  of  your  devices? 

2.  Do  you  know  precisely  how  many  different  configurations  you  have  in  your  environment? 

3.  Can  you  reliably  rebuild  servers  that  are  in  production  (i.e.  "we  can  do  bare-metal  builds")? 

4.  Do  you  perform  change  audits  for  all  of  your  production  systems? 

5.  Are  production  and  development  systems  clearly  separated? 

6.  How  do  you  ensure  that  the  staging  environment  matches  the  pre-production  environment  before 
deploying  builds  into  production? 

7.  Can  you  capture  the  known  good  state  or  "golden  build"  as  part  of  the  release  management  process? 

8.  Do  you  have  confidence  that  the  deployed  systems  match  the  golden  build? 

9.  Do  you  have  a  library  of  automated  build  systems  for  all  critical  devices  (i.e.  repeatable,  automated 
processes  that  can  provision  all  critical  systems)? 

Resolution  Processes 

1.  Is  change  regardless  of  source  the  root  cause  of  most  of  your  remediation  efforts? 

2.  Is  the  longest  part  of  your  repair  cycle  spent  diagnosing  what  is  wrong? 

3.  Can  you  quickly  detect  what  changed  on  systems  during  problem  resolution? 

4.  Can  you  perform  precision  rollbacks  when  undesired  change  is  detected? 

5.  During  remediation,  can  you  see  all  authorized  work  orders  pertaining  to  a  target  system? 

6.  During  remediation,  can  you  see  all  previous  work  tickets  to  learn  what  has  caused  failures  on  your  system 
in  the  past? 

7.  Do  you  track  the  change  success  rate? 

8.  Do  you  have  a  process  that  maps  all  changes  to  a  valid  business  purpose  or  an  authorized  work  order? 

9.  Have  you  identified  a  set  of  change  owners  for  all  critical  systems? 

10.  Do  you  have  a  list  of  repeat  offenders  who  circumvent  change  management  policies? 

From  Denial,  Acceptance  and  Problem  Solving 

If  you  are  like  most  IT  operation  organizations,  you  may  have  uncovered  some  issues  with  this  short 
questionnaire.  Do  not  worry,  because  these  are  common  problems  that  ITIL  was  designed  to  solve. 
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|  Appendix  C  Focusing  Efforts  With  An  Integrity 
Management  Capability  Assessment  (IMCA) 

The  IMCA  exam  leverages  lessons  learned  from  ITIL,  BS  15000  and  high  performing  organizations  to  measure 
an  organization  in  four  key  ITIL  process  areas:  Release  Processes,  Control  Processes,  Resolution  Processes  and 
the  Security  Management  portion  of  the  Service  Design  &  Management  Processes  domain.  The  exam  is 
conducted  during  a  one-hour  interview  session  wherein  numeric  scores  are  assigned.  The  end  result  is  a 
detailed  report  approximately  20  pages  long  that  identifies  strengths  and  weaknesses  in  the  areas  mentioned. 
The  questions  asked  in  Appendix  B  are  indicative  of  the  questions  asked  for  the  exam. 

The  principal  benefits  of  an  IMCA  exam  are  that  it  helps  to  bring  unspoken  weaknesses  to  light  and  creates 
executive  level  champions.  Without  an  IMCA,  organizations  can  embark  without  the  initial  knowledge  of 
process  status,  but  this  is  akin  to  taking  a  shotgun  approach  to  potential  root  causes.  The  IMCA  can  create  a 
capability  assessment  that  can  then  be  analyzed  to  accelerate  a  return  on  your  efforts. 

More  information  is  available  at  http://www.itpi.org 
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Appendix  "  A  Glossary  Of  Terms 

The  following  glossary  is  an  attempt  at  identifying  possibly  ambiguous  terms  used  in  this  book.  A  current  and 

more  complete  glossary  is  available  on  the  ITP1  Web  site  http://www.itpi.org. 

•  Availability:  Typically  stated  as  the  ratio  of  the  time  that  the  system  is  operating  within  acceptable  bounds 
divided  by  the  total  possible  time. 

•  Business  As  Usual  (BAU)  Changes:  These  are  changes  that  are  regularly  made  during  the  course  of  business 
with  readily  known  risks  and  outcomes  are  readily  known. 

•  Change  Advisory  Board  (CAB):  A  defined  group  of  stakeholders  with  vested  interests  in  the  system  in 
question  who  are  able  to  weigh  the  risks  and  benefits  of  change  while  maintaining  proper  communication. 

•  Change  Advisory  Board/Emergency  Committee  (CAB/EC):  When  there  is  an  urgent  or  emergency  change 
and  the  entire  CAB  cannot  be  convened,  this  is  a  defined  smaller  group  of  stakeholders  who  can  review  the 
change  request  and  make  a  proper  decision  as  to  implementation.  Their  decision  should  then  be  reviewed 
when  the  CAB  next  convenes. 

•  Change  Success  Rate  (CSR):  The  ratio  of  successful  changes  to  total  changes.  This  metric  shows  the  relative 
effectiveness  of  the  change  management  process. 

•  Configuration  Drift:  The  tendency  for  configurations  to  change  over  time.  For  example,  a  server's 
configuration  is  most  certainly  known  when  it  is  first  released.  As  time  goes  on,  patches  get  applied,  and 
there  is  human  intervention.  As  these  changes  accumulate,  undocumented  changes  may  have  occurred  and 
the  actual  configuration  has  thus  "drifted"  away  from  the  known  configuration. 

•  Configuration  Item  (Cl):  One  discrete  build  that  is  tracked.  It  may  be  a  base  component  that  can  not  be 
further  divided  or  an  assembly  made  up  of  other  configuration  items.  CIs  can  be  hardware,  software, 
documentation  or  a  combination  thereof. 

•  Configuration  Management  Database  (CMDB):  A  system  used  to  track  configuration  items,  requests  for  change, 
work  orders,  errors,  relationships,  etc.  The  definition  is  often  nebulous  as  the  exact  implementation  varies  across 
organizations.  Fundamentally,  it  is  the  core  system(s)  that  tracks  all  activities  including  service  levels. 

•  Defense-in-Depth:  A  security  strategy  of  using  many  layers  of  defense  as  opposed  to  relying  on  fewer  layers, 
perhaps  even  just  a  single  layer.  The  thought  process  is  that  by  using  layers,  each  provides  an  additional  level 
of  security  should  the  preceding  barrier  be  breached. 

•  Definitive  Software  Library  (DSL):  A  repository  of  authorized  software  that  is  secure  and  has  version 
control.  Software  may  only  be  added  or  removed  from  the  library  through  formal  processes. 

•  Detective  Control:  Processes  or  systems  that  review  records  to  determine  activity.  In  IT,  a  change  monitoring 
and  reporting  system  that  reviews  configurations  against  known  standards  on  a  periodic  basis  and  reports 
observed  changes  is  an  example  of  a  detective  control.  Likewise,  a  manual  review  of  a  log  file  looking  for 
anomalies  is  an  example  of  a  detective  control. 

•  Diff:  To  create/detect  a  delta  or  "difference"  between  two  items.  Take  lists  "ABC"  and  "CDE".  The  diff  is 
"ABDE"  as  they  are  unique  to  each  list  while  "C"  is  common  in  both  lists. 
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•  Domain  Name  Server  (DNS):  A  server  application  used  to  map  hostnames  to  Internet  Protocol  (IP) 
addresses. 

•  Dynamic  Host  Configuration  Protocol  (DHCP):  A  protocol  that  automatically  communicates  network 
configuration  settings  from  a  central  host  to  distributed  assets,  which  are  configured  to  use  the  protocol  to 
obtain  their  Internet  addresses,  domain  name  servers,  gateways,  etc. 

•  Forward  Schedule  of  Change  (FSC):  A  schedule  that  contains  details  of  all  changes  and  their  proposed 
implementation  dates.  Items  are  added  to  it  through  the  approved  change  control  process. 

•  Integrity  Management  Capability  Assessment:  See  Appendix  C. 

•  ITPI  Community  of  Practice  List  (ICOPL):  An  email  list  maintained  by  the  ITPI  organization  to  facilitate 
knowledge  transfer. 

•  Information  Technology  Infrastructure  Library  (ITIL):  A  collection  of  best  practices  codified  in  seven 
books  by  the  Office  of  Government  Commerce  in  the  UK.  http://www.ogc. gov.uk/index. asp?id=2261 

•  Information  Technology  Process  Institute  (ITPI):  A  not-for-profit  organization  engaged  in  three  principle 
areas  of  activity:  Research,  Benchmarking  and  the  Development  of  prescriptive  guidance  for  practitioners 
and  business  executives. 

•  Mean  Time  Between  Failures  (MTBF):  The  average  time  between  failures  of  the  asset. 

•  Mean  Time  To  Repair  (MTTR):  The  average  time  it  takes  to  restore  service  once  an  asset  has  failed  or 
dropped  below  acceptable  service  levels. 

•  Request  For  Change  (RFC):  A  formally  submitted  document,  or  electronic  record,  that  identifies  the  relevant 
information  surrounding  the  desired  change. 

•  Revision  Control  System  (RCS):  An  application  that  tracks  versions  of  files  or  potentially  another  form  of 
data  through  the  use  of  access  and  check-in/check-out  controls.  Depending  on  the  system,  a  baseline  with 
subsequent  differentials  may  be  tracked  or  each  new  version  is  stored  and  assigned  a  unique  ID. 

•  Shelf  Life/Release  Shelf  Life  (RSL):  Defines  how  long  a  build  will  remain  viable  before  becoming  obsolete. 
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CMDB  Table  Structures 

Assembling  a  CMDB  can  be  done  in  many  ways.  The  following  table  represents  key  fields  to  be  included  in  any 
CMDB  at  the  Cl  master  level: 

The  following  flags  can  be  used  to  track  the  current  status  of  each  Cl  entry: 


1  Attribute 

Description 

Cl  ID 

A  unique  name  identifier  by  which  the  Cl  can  be  identified 
(Company  Name-Location-CI  Type) 

Cl  ID  Number 

A  unique  number  generated  by  the  CMDB 

Cl  Description 

A  description  of  the  Cl 

Cl  Category 

General  category  of  the  Cl 

Owner  Responsible 

Name  of  person  responsible  for  this  Cl 

Customer 

Company  name  of  customer 

Date  Acquired 

Date  the  organization  took  ownership 

Status 

Is  the  component  currently  registered,  accepted,  under 
development,  installed,  withdrawn,  etc. 

Next  Maintenance 

Window 

Date  of  next  scheduled  maintenance  (if  applicable) 

Make 

Manufacturer 

Model 

Model  Name 

Model  Number 

Model  Number 

Part  Number 

Hardware  part  number 

Serial  Number 

Hardware  serial  number 

License  Number 

Software  license  number 

Version  Number 

Software  version  number 

Source  Supplier 

Who  provided  the  component? 

Relationship 

Parent/Child;  Cl  is  connected  to  another  Cl;  Cl  is  resident  in 
another  Cl;  Cl  using  another  Cl 

Relationship  Number 

Cl  IDs  are  used  to  create  the  Relationship  Number. 

Location 

The  physical  location  of  the  device.  Data  center-rack 
location-rack  unit  ID 

Ticket  Numbers 

The  ticket  number  of  incidents,  problems,  and  change 
requests  related  to  this  Cl 

Description 
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The  following  flags  can  be  used  to  track  the  current  status  of  each  Cl  entry: 


1  Status  Flag 

Cl  has  been  ordered  from  a  vendor  but  has  not  yet  arrived  and 
therefore  cannot  be  registered. 

Registered 

Cl  has  been  received  and  fully  identified  in  the  MDB  database. 

Accepted 

Cl  has  been  accepted  by  the  Cl  owner.  This  designation  means  that  the  Cl 
process  owner  has  verified  that  the  Cl  meets  the  specifications  that  were  called 
out.  This  is  a  verification  that  a  Quality  Assurance  process  has  occurred. 

The  process  owner  during  registration  is  the  "Owner  Responsible." 

Development 

Cl  is  in  the  development  environment. 

Testing 

Cl  is  in  the  testing  environment. 

Installed 

Cl  is  in  the  production  environment. 

Under  Change 

Cl  is  in  the  process  of  being  changed. 

DSL 

Cl  is  part  of  the  Definitive  Software  Library. 

DHL 

Cl  is  part  of  the  Definitive  Hardware  Library. 

Archived 

Cl  is  archived  and  under  the  control  of  Storage  Management. 

Obsolete 

Cl  is  obsolete. 

Missing 

Cl  is  missing. 

Stolen 

Cl  was  stolen. 

Description 
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Software  List 


Automated  build  systems 

AIX 

NIM,  URL:  http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/sg245524.html70pen, 
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Solaris 
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Windows 

InstallShield  AdminStudio,  URL:  http://www.installshield.com/products/adminstudio/,  last  visited 
March  7,  2004. 

Change  monitoring  and  reporting 

Tripwire  for  Servers,  Tripwire  for  Network  Devices,  URL:  http://www.tripwire.com,  last  visited 
March  7,  2004. 

Ticketing  systems 

Best  Practical  RT/RTIR  (Request  Tracker,  Request  Tracker  for  Incident  Response),  URL: 
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HP  OpenView  Service  Desk,  URL:  http://managementsoftware.hp.com/products/sdesk/index.html, 
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Appendix  G 1  High-Performing  IT  Organizations:  What  You  Need  to  Change  to  I 
Become  One16 

i 

IT  is  being  challenged  on  many  fronts,  from  cost  containment,  business  alignment,  compliance,  competitive 
pressures  in  managing  outsourced  IT  services,  and  security.  Many  experienced  IT  practitioners  confronted  by 
this  potentially  staggering  array  of  challenges  will  point  out  that  the  solution  to  virtually  all  these  issues  is  more 
repeatable  IT  processes  and  effective  controls.  However,  merely  understanding  this  does  not  necessarily  equate  ’ 
to  an  effective  plan  to  solve  the  problems,  and  may  create  more  questions  than  answers.  To  simplify  the  problem, 

Dr.  Eliyahu  Goldratt,  creator  of  the  Theory  of  Constraints,  articulates  three  simple  questions  that  must  have  1 
credible  answers:  What  do  I  need  to  change,  what  should  I  change  to,  and  how  do  I  cause  the  change?  ] 

Finding  answers  to  those  three  questions  has  been  an  area  of  passion  for  Gene  since  1999.  He  has  been  research-  ' 
ing  high-performing  IT  operations  and  security  organizations,  attempting  to  understand  what  makes  them  so  f 
different  than  typical  IT  organizations,  as  well  as  studying  how  organizations  have  accomplished  the  1 
transformations  that  take  them  from  being  merely  average  to  best  in  class.  Along  this  journey,  Gene  started  1 
working  with  other  organizations  that  are  also  interested  in  these  issues,  such  as  SANS,  the  IT  Process  Institute  1 
(ITPI),  the  Institute  of  Internal  Auditors  (IIA),  and  most  recently,  the  Software  Engineering  Institute  (SEI)  with  1 
Julia  Allen.  In  particular,  the  collaboration  between  the  ITPI  and  SEI  has  yielded  some  extremely  promising  ’ 
results,  both  in  characterizing  high-and  low-performing  IT  organizations,  the  key  differences  in  their  belief  ] 
systems,  and  the  necessary  components  to  achieve  an  organizational  transformation  from  low-  to  high-performer.  ; 

In  this  article,  we  will  discuss  two  areas  of  research  that  we  believe  are  foundational  for  answering  the  question  1 
of  what  IT  organizations  typically  need  to  change  and  what  they  need  to  change  it  to.  We  will  present  a  working 
definition  of  what  characterizes  a  high-performing  IT  organization,  and  then  discuss  the  key  differences  in  the  1 
belief  systems  between  them  and  more  typical  IT  organizations  in  three  areas  of  pain:  patch  management,  1 
proliferation  of  IT  management  scorecards,  and  managing  outsourced  IT  services. 

) 

Lastly,  to  help  answer  the  question  of  how  to  cause  the  change,  we  will  describe  the  publicly  available  Visible  ; 
Ops  methodology,  which  captures  how  IT  organizations  have  transformed  into  high-performers  in  a  way  that 
is  can  be  accomplished  in  four  steps,  each  which  is  a  finite  project  and  returns  more  value  back  than  was 
invested.  We  will  also  describe  the  ITPI  Community  of  Practice  Listserv,  and  the  upcoming  VEESC 
benchmarking  study.  We  conclude  the  article  with  a  call  to  action  and  an  active  solicitation  for  feedback  in 
participation  in  creating  this  community  of  practice  for  high-performing  IT  organizations. 

Key  Characteristics  of  High-Performing  IT  organizations 

Since  1999,  after  studying  the  IT  processes  of  hundreds  of  organizations,  it  started  becoming  clear  to  Gene  that 
a  handful  of  them  stood  out  as  somehow  different  from  the  others  in  some  notable  way.  He  started  keeping  a 
list  of  these  organizations,  at  that  time  informally  called  "Gene's  list  of  people  with  amazing  kung  fu."  In  2000, 
Gene  started  working  with  Kevin  Behr,  CTO  of  one  of  these  unusual  organizations,  and  they  started  a  more 

i 

16  Adopted  from:  Allen,  Julia  and  Kim,  Gene.  "High-Performing  IT  Organizations:  1. 

What  You  Need  to  Change  to  Become  One."  The  original  article  was  previously  published  and  is  available  at 
http://www.bettermanagement.com/Library/Library.aspx?a=13&LibraryID=9429.  Reprinted  with  permission  from  BetterManagement.com 
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systematic  analysis  of  what  was  common  to  these  organizations,  which  they  renamed  the  "best  in  class  IT 
operations  and  security  organizations."  In  2003,  Julia  Allen  from  the  SEI  actively  joined  this  effort,  which 
resulted  in  a  remarkable  event  in  October  2003  at  Carnegie  Mellon  University  called  the  Best  In  Class  Security 
and  Operations  Roundtable  (BIC-SORT).  Among  the  stated  goals  were  to  "begin  to  build  an  executive-level 
community  of  practice  for  IT  (information  technology)  operations  and  security,  with  a  common  sense  of 
purpose  and  a  desire  to  influence  other  relevant  and  connected  communities  of  practice;  and  to  better  capture 
and  articulate  the  relevant  bodies  of  knowledge  that  enable  and  accelerate  IT  operational  and  security  process 
improvement."  Since  then,  we  have  been  actively  processing  and  synthesizing  the  data  we  collected. 

Based  on  our  analysis,  we  have  created  the  following  working  definition  of  high-performing  IT  organizations: 
They  are  effective  and  efficient  and  they  succeed  in  applying  resources  to  accomplish  their  stated  business 
objectives  with  little  to  no  wasted  effort.  These  organizations  have  evolved  a  system  of  process  improvement 
as  a  natural  consequence  of  their  business  demands.  They  regularly  implement  formal,  repeatable  and  secure 
operational  processes. 

Results  of  informal  benchmarking  indicate  that  in  these  best-in-class  IT  organizations,  IT  operations  and 
security  work  together  to  create  higher  service  levels  (e.g.  as  measured  by  mean  time  to  repair,  mean  time 
between  failure);  higher  percentage  of  planned,  scheduled  work  relative  to  unplanned  work;  unusually 
efficient  cost  structures  (e.g.  as  measured  by  server  to  system  administrator  ratios);  productive  working 
relationships  with  management  and  peers;  and  smoother  audits.  Furthermore,  they  have  more  timely 
identification  and  resolution  of  security  incidents,  the  earliest  integration  of  information  security 
requirements  in  the  service  delivery  lifecycle,  and  the  ability  to  quickly  return  to  a  reliable  and  trusted 
operational  state.  And  perhaps  most  admirably,  these  organizations  devote  increasingly  more  time  and 
resources  to  strategic  issues,  having  mastered  tactical  concerns. 

The  high-performing  organizations  desire  to  detect  production  variances  early  so  they  can  fix  problems  in  a 
planned  manner  and  where  the  repair  costs  are  lowest  and  have  the  least  impact.  They  value  repeatable  and 
verifiable  processes  and  use  controls  to  improve  efficiency  and  effectiveness.  And  because  these  organizations 
use  controls  to  improve  their  own  operation,  life  is  much  easier  for  auditors  who  evaluate  operational  risk 
based  on  the  presence  of  effective  and  verifiable  preventive,  detective,  and  corrective  controls.  In  other  words, 
the  controls  aren't  there  just  because  auditors  asked  for  them,  but  because  they  are  used  to  improve  daily 
operations!  As  a  result,  high-performing  organizations  require  considerably  less  effort  to  meet  management 
and  audit  expectations. 

To  achieve  these  characteristics,  several  key  performance  metrics  are  essential  to  this  level  of  performance:  they 
have  the  highest  change  success  rate  (typically  over  98%),  highest  effective  rate  of  change  (sometimes  making 
over  1000+  successful  changes  per  week),  highest  level  of  mastery  of  production  infrastructure  (achieved  by 
low  configuration  counts  and  low  configuration  variance),  and  highest  ratio  of  staff  dedicated  to  pre- 
production  activities  (achieved  by  release  management  processes,  pre-production  testing,  etc.). 

Surprisingly,  we  found  that  all  of  the  high-performing  IT  organizations  had  independently  developed  virtually 
the  exact  same  processes  to  achieve  these  results.  They  shared  similarities  in  three  key  process  areas,  which  we 
will  describe  in  the  parlance  of  ITIL  (IT  Infrastructure  Library):  they  had  a  "culture  of  causality"  that  ensured 
all  production  problems  ruled  out  change  as  early  as  possible  in  the  repair  cycle  (resolution  processes),  they 
had  a  "culture  of  change  management"  embedded  in  the  way  all  work  is  done  (control  processes),  and  they 
moved  as  many  production  changes  through  a  pre-production  process  that  orchestrated  changes  with  the 
production  environment  (release  processes). 
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The  high-performing  organizations  all  implemented  virtually  the  same  procedures  in  these  three  1TIL  process 
areas,  which  form  the  minimal  closed-loop  that  generates  metrics  that  allow  continual  process  improvement. 
These  procedures  and  processes  are  described  in  the  Visible  Ops  methodology  in  detail,  published  by  the  ITPI 
(http://wss-sv.itpi.org/home/visibleops.php). 

Belief  System  Differences  Between  High-  And  Low-  Performing  IT  Organizations 

Given  the  fact  that  high-performing  IT  organizations  exist,  what  prevents  low-performing  organizations  from 
becoming  high-performers,  given  the  promise  of  a  better  way?  Understanding  why  this  was  so  became  one  of  the 
main  areas  of  activity  after  the  BIC-SORT  event.  Julia  Allen,  Kesin  Behr,  and  Gene  Kim  from  the  ITPI  and  SEI  have 
been  synthesizing  the  captured  list  of  key  areas  of  pain  and  promise  from  the  participating  organizations  during 
BIC-SORT.  Our  goal  svas  to  create  a  taxonomy  of  pains,  find  any  cause-effect  relationships  and  root  causes,  and 
understand  what  belief  systems  that  preserved  the  status  quo  for  the  low-performers. 

In  the  BIC-SORT,  we  captured  almost  one  hundred  specific  areas  of  pain,  such  as  the  challenges  of  keeping  up 
with  security  patches,  the  massive  efforts  required  to  do  effective  audits  of  business  peers,  and  so  forth.  Of 
these,  we  chose  to  analyze  three  of  the  most  acute  of  the  Listed  pains:  keeping  up  with  patching,  dealing  with 
the  proliferation  of  management  scorecards,  and  management  of  outsourced  IT  services. 

To  analyze  these  problems,  we  used  a  technique  pioneered  by  Dr.  Eliyahu  Goldratt  called  the  Theory  of 
Constraints  Thinking  Tools,  specifically  problem  clouds  and  current  reality  trees  (for  more  information,  see 
http://wss-w.thedecalogue.com/Tools/crt.htm).  The  goal  was  to  understand  the  causal  factors  and  beliefs  that 
led  to  the  high-  and  low-performing  behaviors,  and  then  finally,  to  find  any  commonalities  among  the  three 
pain  areas.  What  we  found  was  illuminating. 

Volume  of  Security  Patches 

An  area  of  pain  articulated  by  many  of  the  participants  at  the  BIC-SORT  was  the  volume  of  urgent  patches 
needing  to  be  applied  to  infrastructure,  resulting  from  the  constant  stream  of  new  security  vulnerabilities,  and 
the  need  to  find  an  effective  solution  to  managing  patches. 

In  the  low-performers,  this  activity  was  characterized  as  ad  hoc,  chaotic,  and  urgent.  Announcement  of  the 
availability  of  a  patch  to  address  a  critical  security  vulnerability  would  lead  to  widespread  chaos  and 
disruption,  often  resulting  in  massive  amounts  of  unplanned  work  at  the  expense  of  planned  work.  Worse, 
even  successfully  deploying  the  patch  would  often  cause  unintended  consequences,  such  as  servers  becoming 
non-functional  or  even  non-booting. 

In  contrast,  the  high-performers  addressed  patching  as  a  predictable  and  planned  activity,  treating  them  as  just 
another  change.  Announcement  of  critical  patches  would  result  in  merely  adding  the  patch  to  the  release 
engineering  candidate  queue,  where  it  could  be  evaluated,  tested  and  integrated  into  an  already  scheduled 
deployment.  The  absence  of  urgency  and  a  well-defined  process  for  integrating  changes  leads  to  a  much  higher 
change  success  rate.  Interestingly,  virtually  all  of  the  high-performers  apply  patches  much  less  frequently  than 
the  low-performers,  perhaps  by  one  or  two  orders  of  magnitude! 

Proliferation  of  IT  Management  Scorecards 

BIC-SORT  attendees  also  listed  the  proliferation  of  IT  management  "scorecards"  and  other  management  and 
assessment  instruments  as  another  area  of  pain.  We  also  threw  into  this  category  all  the  various  industry  compli¬ 
ance  requirements,  ranging  from  Sarbanes-Oxley  Section  404,  Gramm-Leach-Bliley,  E1IPAA,  and  so  forth. 
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In  the  low-performers,  this  activity  was  characterized  by  having  to  look  to  external  sources  and  authorities  for 
the  desired  behaviors  and  measurements.  The  absence  of  a  strong  internal  IT  management  framework  and 
belief  system  might  lead  to  adopting  a  "scorecard  du  jour,"  or  worse,  multiple  external  scorecards 
simultaneously  that  conflict  with  each  other.  This  would  lead  to  more  work  for  the  organization,  and  excess 
retrofitting  to  deal  with  the  necessary  process  and  organizational  gymnastics.  Worse,  executive  turnover  might 
result  in  switching  scorecards,  which  repeats  the  entire  chaos  cycle. 

In  contrast,  the  high-performers  have  their  own  clearly  defined  performance  goals  and  desired  characteristics. 
If  the  need  to  conform  to  an  external  scorecard  or  regulatory  requirement  materializes,  they  assign  a  small 
team  to  demonstrate  traceability  to  it.  Consequently,  they  have  a  lower  cost  of  developing,  sustaining  and 
documenting  controls,  a  better  posture  of  audit  and  compliance,  and  have  little  need  to  look  externally  for 
authorities  to  tell  them  how  they  need  to  operate. 

Managing  Outsourced  IT  Services 

The  last  area  of  pain  we  analyzed  was  the  challenge  of  managing  outsourced  IT  services.  Any  challenges  with 
IT  are  inherently  made  more  complex  when  these  services  are  provided  by  an  outside  provider  instead  of  an 
employee:  corrective  actions  may  have  contractual  implications,  the  scope  of  corrections  may  be  constrained 
by  the  service  level  agreement,  and  so  forth. 

In  the  low-performers,  there  is  often  a  real  desire  to  transfer  the  IT  risk  and  responsibilities  to  someone  else, 
especially  if  management  perceives  an  absence  of  internal  skills  to  meet  the  business  objectives.  However, 
when  IT  functions  are  outsourced,  such  functions  rapidly  become  out  of  sight  and  out  of  mind,  until  the 
organization  finds  that  it  is  unable  to  control  and  attest  to  the  controls  of  the  service  provider.  The 
organization  then  discovers  that  it  may  have  inadvertently  exacerbated  the  challenges  by  outsourcing,  but 
unfortunately,  "re-insourcing"  the  services  may  no  longer  be  an  option. 

In  contrast,  the  high-performers  manage  outsourced  IT  services  just  like  any  other  business  unit  or  project.  They 
understand  the  unique  positive  and  negative  challenges  of  fulfilling  IT  projects  or  services  by  an  external  party. 
As  a  result,  they  tend  to  develop  more  bullet-proof  service  level  agreements  to  proactively  get  better  service 
and  create  avenues  for  future  corrections  from  the  service  provider. 

Common  Root  Causes  For  Preservation  Of  the  Status  Quo 

After  analyzing  the  three  areas  of  pain,  we  started  looking  for  common  patterns  and  root  causes  that  led  to  the 
preservation  of  the  status  quo  in  the  low-performers,  despite  the  clear  promise  of  achieving  the  characteristics 
of  the  high-performers.  We  found  five  areas  of  root  cause. 

1.  The  absence  of  explicit  articulation  of  current  state  and  desired  state  hides  or  obscures  the 
amount  of  pain 

Often,  management  will  conclude  that  the  current  state,  along  with  all  the  companion  pains,  is  tolerable. 
These  organizations  may  articulate  a  litany  of  pains  and  frustrations,  but  in  the  absence  of  being  able  to 
quantify  the  pain,  may  decide  without  that  it  probably  does  not  hurt  enough  yet  to  warrant  any  corrective 
action.  This  may  be  because  of  a  sincere  belief  that  the  pain  is  not  high  enough  yet,  or  it  may  be  the  following: 

2.  A  culturally  embedded  belief  that  control  is  not  possible 

Often,  management  may  not  know  that  there  is  an  alternative,  believing  that  the  control  is  not  possible  due 
to  its  nature  (i.e.  "IT  operational  and  security  issues  are  like  the  weather:  there  is  nothing  we  can  do  about  it, 
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and  that  bad  things  happen  to  us,  just  like  rain  or  hurricanes"),  control  is  not  possible  due  to  business  needs  ; 
(i.e.  "my  business  environment  is  too  dynamic  to  accommodate  bureaucratic  processes  or  controls"),  or  maybe  j 
even  deliberate  abdication  of  responsibility. 

3.  Rewards/reinforcements  for  personal  heroics  vs.  repeatable,  predictable  discipline 

Often,  there  may  be  a  culture  or  a  hidden  reward  system  that  encourages  heroics  and  a  "cowboy  culture."  For  1 
instance,  one  person  may  work  throughout  the  night  for  an  entire  weekend  fighting  a  fire  and  get  rewarded  as 
the  hero  who  saved  the  day.  What  is  overlooked  is  that  if  one  person  can  save  the  entire  boat,  one  person  can  1 
probably  sink  it,  too.  In  these  organizations,  implementing  effective  processes  and  controls  may  be  resisted  or 
actively  ejected,  almost  as  an  immune  system  would  resist  an  unknown  and  foreign  object. 

< 

4.  Continued  argument  that  IT  operations  and  security  are  different  than  other  business  investments  or  ( 

projects  i 

Often,  there  may  be  a  view  that  IT  is  different  than  other  business  functions  or  projects,  thus  leading  to  need  1 
to  determine  the  "business  alignment  of  IT."  Worse,  IT  may  be  operating  as  a  silo,  but  here  may  be  a  separate  1 

C 

security  silo  inside  it!  There  is  a  common  belief  that  ongoing  security  can  exist  outside  the  scope  of  IT 
operations.  While  security  requirements  certainly  exist  outside  of  the  IT  context,  security  controls  must  be 
embedded  into  IT  processes  so  that  they  are  jointly  owned  by  both  the  IT  and  security  organizations.  When 
the  two  organizations  do  not  have  defined  roles  where  they  are  collectively  solving  common  business 
objectives,  blame-games  and  finger-pointing  for  failures  can  cause  a  downward  spiral. 

5.  A  desire  for  a  technical  solution,  which  is  easier  to  justify  and  implement  than  people  and  process 
improvements 

Often,  because  of  their  backgrounds,  IT  management  values  automation  and  technology  over  repeatable 
processes  and  controls.  In  the  absence  of  properly  functioning  processes  and  controls,  the  massive  deployment 
of  security  technology  solutions  invariably  results  in  the  staggering  capability  to  automatically  perform 
devastating,  irreversible  IT  operational  changes  in  mere  seconds,  resulting  in  potentially  monumental  episodes 
of  unplanned  work  and  chaos  for  the  entire  organization.  Combined  with  the  previous  root  cause,  this  factor 
creates  the  kindling  for  an  extremely  fast  and  accelerated  downward  spiral. 

The  entirety  of  these  findings  is  available  in  a  report  published  by  the  SEI  as  follows:  Allen,  Julia;  Behr,  Kevin; 
Kim,  Gene  et  al.  Best  in  Class  Security  and  Operations  Round  Table  Report  (CMU/SEI-2004-SR-002).  Pittsburgh,  PA: 
Software  Engineering  Institute,  Carnegie  Mellon  University,  March  2004.  Copies  of  the  report  are  available 
upon  request. 

Summary  and  Call  To  Action 

In  this  article,  we  explored  three  critical  questions  in  the  context  of  solving  the  most  common  IT  challenges: 
What  do  I  need  to  change,  what  should  I  change  to,  and  how  do  I  cause  the  change? 

By  studying  high-performing  IT  organizations,  the  areas  that  most  often  need  changing  in  lower  performing 
organizations  are  those  with  cultures  that  sustain  a  belief  that  control  is  not  possible,  that  the  absence  of 
controls  have  tolerable  costs,  that  success  of  the  individual  can  outweigh  the  needs  for  success  of  the 
organization,  and  that  somehow  IT  security  and  operations  are  independent  of  each  other.  By  overcoming 
these  incorrect  beliefs,  and  by  implementing  repeatable  processes  in  the  1TIL  process  areas  of  release,  controls 
and  resolution  as  outlined  in  the  Visible  Ops  methodology,  organizations  can  not  only  achieve  a  belief 
transformation,  but  a  performance  transformation  as  well. 
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So,  here  is  our  call  to  action:  Do  you  agree  or  disagree  with  our  definitions  of  high-  and  low-performing  IT 
organizations?  Do  you  have  more  characteristics  that  should  be  added  to  our  list  of  best-in-class  attributes?  If 
so,  please  let  us  know  by  emailing  us  at  genek@tripwire.com  or  jha@sei.cmu.edu. 

Also,  if  you  are  interested  in  any  of  this  work,  please  join  the  ICOPL  mailing  list.  Subscription  information  is 
at  http://www.itpi.org/home/icopl.php. 

About  Julia  Allen 

Julia  Allen  is  a  senior  member  of  the  technical  staff  within  the  Networked  Systems  Survivability  Program  at  the 
Software  Engineering  Institute  (SEI),  Carnegie  Mellon  University  (CMU).  The  CERT®  Coordination  Center  is 
also  a  part  of  this  program.  Allen  is  engaged  in  developing  and  transitioning  enterprise  security  frameworks 
and  executive  outreach  programs  in  information  security,  survivability,  and  resiliency.  Previously,  Allen  served 
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IMCA 


Integrity  Management  Capabilities  Assessment 
What  is  it? 

IMCA  (Integrity  Management  Capabilities  Assessment)  is  a  powerful  benchmarking  tool,  based  on  Information 
Technology  Infrastructure  Library  (ITIL)  best  practices  for  IT  and  the  BS  15000  Code  of  IT  practice.  The 
assessment  was  developed  with  two  primary  goals  in  mind.  First,  to  capture  a  clear  and  detailed  picture  of  an 
organization's  current  strengths,  weaknesses  and  areas  of  risk  with  regards  to  the  integrity  management 
capabilities  required  to  maintain  a  secure,  stable  IT  environment.  Second,  to  present  specific  recommendations 
based  on  an  in-depth  analysis  of  the  unique  operational  challenges  and  IT  environment  being  measured. 

What  is  the  process? 

The  assessment  begins  with  an  interview  process  that  takes  approximately  one  hour.  The  completed 
questionnaire  goes  on  to  produce  an  executive  summary  that  scores  your  operation  in  several  key  ITIL  process 
areas:  Release,  Controls,  Resolution  and  Security.  Your  results  are  then  compared  to  other  companies  in  your 
industry  as  well  as  industry  best  practices,  with  the  final  results  delivered  in  a  comprehensive  document 
(approximately  20+  pages).  This  evaluation  is  used  to  develop  a  recommended  roadmap  that  provides  specific 
process  improvements.  The  improvements  include  the  prescriptive  adoption  of  specific  best  practices  that  are 
relevant  to  your  operation. 

What  results  can  I  expect? 

Have  you  ever  wanted  to  objectively  measure  the  efficiency  of  your  IT  operations?  Many  of  our  clients  have 
expressed  concerns  about  their  ability  to  quantitatively  analyze  the  effectiveness  of  their  IT  operations.  After 
conducting  the  IMCA,  these  organizations  were  able  to  identify  common  "best  in  class"  characteristics  where 
Security,  Operations,  Audit,  and  Management  can  work  together  to  reach  common  objectives. 

Organizations  have  benefited  from  IMCA  through  improved  integration  of  Security  into  Operations,  and  have 
attained  improved  operational  service  levels  and  efficiencies,  as  measured  by: 

•  Improved  Server/SysAdmin  ratios 

•  Lower  Mean  Time  To  Repair  (MTTR) 

•  Higher  Mean  Time  Between  Failures  (MTBF) 

•  Reduced  aggregate  downtime 

•  Decreased  security  risks  with  increased  control 

•  Shortened  provisioning  times 

•  Maximum  change  and  configuration  management  integrity 
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Who  should  be  involved? 


To  ensure  maximum  value  from  the  IMCA  process,  companies  should  plan  to  include  at  least  two  of  the 
following  people: 

•  Vice  President  of  Operations  (or  other  sponsoring  executive) 

•  Line  Operations  Manager 

•  Change  Control  Manager 

•  Representatives  from  the  Security  staff 

It  is  also  recommended  to  assess  a  group  that  all  report  to  the  same  business  area/unit. 

Areas  Evaluated  by  the  Assessment 
Control  Processes: 
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The  assessment  examines  your  organization's  capabilities  relating  to  operational  controls,  ranging  from  the 
ability  to  detect  changes  made  to  infrastructure  to  the  evaluation  of  processes  in  place  for  change  ! 
management.  An  effective  change  management  process  requires  review  of  all  proposed  infrastructure  and 
software  changes  before  they  are  made.  This  review  is  typically  performed  by  stakeholders  such  as  Security, 
Operations,  R&D  as  well  as  Internal  Audit.  The  team  decides  whether  the  proposed  changes  are  appropriate  ' 
and  include  the  necessary  safeguards  to  ensure  continuity  of  service. 

The  assessment  also  examines  configuration  management  practices  that  guarantee  all  critical  software  and  t 
configurations  stored  match  those  running  in  production.  This  ensures  a  known,  good  repository  for  all  ) 
standard  configurations  used  across  the  enterprise,  and  guarantees  that  as  configurations  change,  new  | 
configurations  are  documented  and  supersede  the  older  revisions. 


Resolution  Management  Processes: 

i 

This  part  of  the  assessment  analyzes  the  company's  problem  management  processes.  How  quickly  can  change 
to  infrastructure  be  eliminated  as  the  cause  of  a  problem?  Can  all  changes  be  mapped  to  authorized  work  orders?  > 
Studies  show  that  up  to  eighty  percent  of  all  systemic  outages  are  the  result  of  authorized  employees  making 
changes  to  infrastructure.  We  also  know  that  nearly  eighty  percent  of  the  time  it  takes  to  solve  a  problem  is 
consumed  by  the  process  of  pinpointing  the  location  and  nature  of  the  problem. 


We  also  examine  the  organization's  rollback  capabilities  for  use  in  problem  management.  Here  we  look  at  how 
quickly  staff  can  return  the  infrastructure  to  a  known,  good  state  as  it  existed  before  a  problem  occurred.  This 
allows  quicker  restoration  of  services  when  there  is  an  outage  and  also  allows  senior  staff  to  examine  the 
forensic  evidence  from  the  failure  away  from  the  heat  of  battle. 

Release  Management  Processes 

In  release  management  we  are  looking  for  repeatable  builds  capability  for  any  piece  of  critical  infrastructure  from 
scratch.  This  process  is  related  to  configuration  management  where  the  configuration  is  stored  and  maintained. 
The  ability  to  quickly  recreate  the  last  known  good  configuration  is  critical  to  your  organization's  ability  to 
respond  to  incidents  such  as  disaster,  security  breach,  major  vendor  failure  or  the  outbreak  of  a  virus  or  worm.  It 
is  also  crucial  when  deploying  software  upgrades  and  patches  that  fail  or  display  bugs.  The  assessment  also  looks 
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at  acceptance  processes  for  new  software  and  hardware  to  determine  if  they  are  ready  for  mass  deployment  into 
actual  production  capacity.  This  guards  the  integrity  of  both  the  configuration  management  and  repeatable  build 
processes  by  insuring  that  no  new  infrastructure  or  software  is  deployed  that  can't  be  rebuilt  or  controlled  by 
configuration  management  and  that  hasn't  been  reviewed  by  stakeholders  such  as  Security. 

Security 

The  final  area  we  evaluate  is  security.  Security  is  not  a  task  that  is  done  but  rather  the  end-result  of  many  other 
processes  and  controls  operating  effectively.  Here,  we  examine  your  organization's  ability  to  understand  the 
known  good  state  of  IT  infrastructure,  how  it  is  configured  and  built,  and  how  to  know  if  someone  makes  a 
malicious  change  versus  a  change  made  by  internal  staff.  If  an  undesired  change  is  made,  can  you  rollback  to  a 
known  good  state  instantly  and  provide  forensic  evidence  that  proves  there  was  indeed  malicious  activity?  How 
well  have  all  of  the  critical  pieces  of  infrastructure  been  documented  so  that  recovery  from  a  disaster  is  possible? 
Can  policy  circumvention  be  tracked— is  it  possible  to  tell  when  change  control  policy  has  been  violated?  How 
easy  is  it  for  changes  and  new  infrastructure  to  make  it  in  to  production  without  a  prior  security  review. 

Summary 

In  conclusion,  the  Integrity  Management  Capabilities  Assessment  was  developed  to  assist  organizations  like 
yours  to  successfully  implement  IT  process  improvements  and  industry  best  practices  in  order  to  ensure  the 
safety,  stability  and  predictability  of  Information  Services  across  your  enterprise.  The  assessment  is  brought  to 
you  by  the  Information  Technology  Process  Institute  (ITPI),  a  non-profit  organization  with  the  unique  charter 
to  educate  IT  and  move  effective  information  Technology  Service  Management  into  the  realm  of  responsible 
corporate  governance.  The  ITPI  serves  to  align  actual  practices  with  Best  Practices  and  creates  tools, 
prescriptive  adoption  methods  and  control  metrics  to  facilitate  management  by  fact. 

The  Integrity  Management  Capabilities  Assessment  was  developed  by  the  ITPI  in  partnership  with  IP  Services 
and  Tripwire.  IP  Services  is  a  global,  technical  consulting  organization  whose  focus  is  to  help  customers 
implement  Managed  Enterprise  Network  Operations  Center  (NOC)  Services  and  E-Business  solutions.  Tripwire 
is  the  worldwide  leader  in  integrity  assurance  solutions,  delivering  software  for  IT  security  and  operations  staffs 
so  they  can  immediately  detect  and  pinpoint  undesired  change  to  their  servers  and  network  devices.  In  this 
way,  Tripwire  enables  rapid  recovery,  ensures  the  stability  of  information  services,  and  increases  systems 
availability  as  well  as  IT  staff  productivity. 


Information  Technology  Process  Institute 

2896  Crescent  Avenue,  Suite  104 
Eugene,  Oregon  97408 
Telephone:  (541)  485-4051,  opt.  5 
Email:  IMCA@itpi.org  www.itpi.org 

To  schedule  an  Integrity  Management  Capabilities  Assessment  or  to  request  additional  information,  please  call 
(541)  485-4051. 


IMCA 
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The  value,  effectiveness,  efficiency,  and  security  of  IT  controls:  An  empirical  analysis 

Information  technology  managers  are  confronted  with  a  myriad  of  best-practice  frameworks  for  information 
technology  service  management.  These  frameworks  include  the  Information  Technology  Infrastructure  Library 
(ITIL)  and  the  Control  Objectives  for  Information  and  related  Technology  (COBIT).  Advocates  of  these 
frameworks  promote  the  value  of  these  guidelines  in  achieving  cost  reductions  and  improving  business 
processes.  The  problem  is  that  implementing  these  frameworks  involves  substantial  upfront  costs.  Many 
practitioners  view  them  as  simply  another  level  of  bureaucracy.  The  purpose  of  this  paper  is  to  determine 
empirically  whether  IT  controls  affect  the  value,  effectiveness,  efficiency,  and  security  of  information- 
technology  operations.  We  hypothesize  that  implementation  of  IT  controls  improves  IT  efficiency,  IT 
effectiveness,  IT  security,  and  indirectly,  business  value.  Based  on  prior  research  and  extensive  pilot  testing 
with  high-performing  organizations,  we  are  developing  a  survey  to  test  our  hypotheses.  We  will  then  distribute 
the  survey  to  a  sample  of  Fortune  1000  companies,  government  departments,  and  universities. 

The  benefits  to  you  and  your  organization  of  completing  the  V.E.E.S.C.  survey  of  practice  include: 

•  Seeing  where  they  rank  nationally  and  by  industry  in  terms  of  IT  operational  excellence. 

•  Respondents  will  have  free  access  to  our  overall  IT  ops  excellence  score  calculations  to  continually 
review  and  rate  their  IT  operations. 

•  Respondents  will  see  evidence  of  the  relationship  between  best  practice  IT  controls  and  improved  IT 
performance  and  return  on  IT  investment. 

•  The  results  will  show  the  inter-relationships  between  the  five  BS  15000  process  areas  and  their  relative 
importance  in  determining  IT  performance. 

•  Respondents  will  see  how  to  improve  their  IT  operational  excellence  ranking. 

The  V.E.E.S.C.  benchmarking  survey  is  a  valuable  addition  to  the  information  in  Visible  Ops:  Starting  ITIL  in 
4  Practical  Steps.  Do  you  want  to  participate  or  find  out  more  information?  Please  contact  us  at  veesc@itpi.org 
or  visit  http://www.itpi.org/home/veesc  for  more  information. 
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Information  Technology  Process  Institute 


Please  accept  this  complimentary  copy  of  The  Visible  Ops  Handbook:  Starting  ITIL 
in  4  Practical  Steps.  We  strongly  feel  that  the  methodology  set  forth  in  this  book  can 
help  organizations  that  are  just  implementing  ITIL,  and  can  provide  fresh  insights  to 
groups  further  down  the  ITIL  path. 

If  you  wish  to  order  additional  copies  to  share  with  your  colleagues,  the  Information 
Technology  Process  Institute  (ITPI)  is  offering  the  following  volume  price  discounts 
from  the  standard  $19.95  per  copy  list  price. 


Number  of  Copies 

1  ~  9 
10-49 
50  -  99 
100  -  249 
250+ 


Price  per  Book  (plus  S&H) 

$14.95 

$11.95 

$10.95 

$9.95 

$8.95 


To  obtain  this  promotional  pricing,  please  visit  www.itpi.org/ciomag  to  place  your 
order.  Shipping  and  handling  is  additional. 


About  the  ITPI 

The  Information  Technology  Process  Institute  (ITPI),  a  not  for  profit  organization,  is 

1  engaged  in  three  principle  areas  of  activity:  research,  benchmarking  and  the 
development  of  prescriptive  guidance  for  practitioners  and  business  executives.  The 
ITPI  has  collaboration  agreements  in  place  with  research  organizations  such  as  the 
Software  Engineering  Institute  at  Carnegie  Mellon  University  and  the  Decision 
Sciences  program  at  the  University  of  Oregon.  We  are  currently  developing  the 
necessary  guidance  that  solves  the  common  objectives  of  IT  Security,  Corporate 
Governance,  Audit  and  Operations.  Through  research,  development  and 
benchmarking,  the  ITPI  creates  powerful  measurement  tools,  prescriptive  adoption 
methods,  and  control  metrics  to  facilitate  management  by  fact. 


You  can  help  us  by  becoming  involved  with  our  efforts  through  subscription  and 
participation  in  organization's  various  activities.  There  are  a  number  of  subscription 
models  available  and  the  three  most  apt  to  apply  are: 

Consultants  -  This  is  aimed  at  people  providing  professional  services  to 
organizations,  and  offers  15  free  copies  of  The  Visible  Ops  Handbook  plus  discounts 
on  books,  events  and  Integrity  Management  Capability  Assessment  (IMCA)  exams. 

Corporations  -  This  level  is  for  corporations  who  are  leveraging  best  practices 
internally.  It  includes  five  free  copies  of  The  Visible  Ops  Handbook  plus  discounts  on 
books,  events  and  IMCA  exams. 

Institutions  -  This  is  ideal  for  nonprofits,  associations,  etc.  It  includes  three  copies 
of  The  Visible  Ops  Handbook  plus  discounts  on  books,  events  and  IMCA  exams. 


Please  visit  www.itpi.org/home/membership.php  for  information  on  these  and 
additional,  levels.  As  mentioned,  in  addition  to  the  subscriptions,  there  are  two 
projects  that  are  in  process  that  you  may  find  of  interest  and  wish  to  become  involved 
with: 

Valuing  the  Effectiveness,  Efficiency,  and  Security  of  IT  Controls  (VEESC)  -  We 
hypothesize  that  implementation  of  IT  controls  improves  IT  efficiency,  IT 
effectiveness,  IT  security,  and  indirectly,  business  value.  Based  on  prior  research  and 
extensive  pilot  testing  with  high-performing  organizations,  we  are  developing  a 
survey  to  test  our  hypotheses.  We  will  then  distribute  the  survey  to  a  sample  of 
Fortune  1000  companies,  government  departments,  and  universities. 

Visible  Ops:  Positive  Control  Environment  (PCE)  -  We  are  working  on  a 
methodology  to  assist  organization  in  their  assessment  of  control  needs  and 
implementation.  PCE  will  leverage  elements  of  Visible  Ops,  ITIL  and  COBIT  to  give 
the  reader  prescriptive  guidance  in  the  establishment,  or  refinement,  of  a  control 
framework  in  IT  that  not  only  meets  regulatory  requirements  but  creates  a  foundation 
to  enable  process  improvement. 


For  more  information,  please  contact  us  at: 

Information  Technology  Process  Institute 

2896  Crescent  Avenue,  Suite  104 
Eugene,  Oregon  97408 

Telephone:  (541)  485-4051,  Fax:  (541)  485-8163 
Email:  IMCA@itpi.org 
Web:  www.itpi.org 


IP  Services  offers  world  class  management  of  business  support 
services  including  revenue  generating  e-business  infrastructure.  We 
apply  repeatable  and  nteasureable  operational  processes  that  control 
your  critical  assets  in  order  to  maximize  availability,  satisfy  compliance, 
and  reduce  cost.  Visit  us  at  www.tcpipservices.com 
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